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ABSTRACT 



A method and apparatus for extracting three-dimensional 
(3-D) data from a target object is disclosed. A plurality of 
markers are formed on the object. A plurality of images are 
captured of the object. A first point is designated from a 
marker from one of the images and a line equation corre- 
sponding to the first point is determined. A second point in 
a marker in another image corresponding to the first point is 
determined and a second line equation corresponding to the 
second point is determined. The intersection of the two line 
equations is determined. 

18 Claims, 12 Drawing Sheets 
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METHOD AND APPARATUS FOR THE 
PROCESSING OF STEREOSCOPIC 

ELECTRONIC IMAGES INTO 
THREE-DIMENSIONAL COMPUTER 
MODELS OF REAL-LIFE OBJECTS 5 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to three- 10 
dimensional (3-D) models and, in particular, to a method and 
apparatus for processing stereoscopic images into 3-D data 
of objects. 

2. Description of the Related Art 

Creating three-dimensional (3-D) models of objects allow 15 
the models to be viewed from many different angles unlike 
two-dimensional (2-D) models that may only be viewed 
from one angle. One method of creating 3-D models is to 
take a multitude of images of real objects from different 
positions and exploit the differences in the objects* projec- 20 
tion. These multitude of images may be suitable to produce 
3-D models of the object. Once a 3-D model is produced, the 
3-D model could be placed in a virtual world and may be 
shared with others, much like photos or TV. In particular, the 
commercial viability of 3-D models is remarkably pro- 25 
nounced in the entertainment, advertisement, and simulation 
industries. 

The desirability and advantages of creating 3-D models of 
objects are readily understood, and many tools have been 
developed to produce 3-D models. For example, 3-D soft- 30 
ware such as 3D Studio MAX from Kinetix, San Francisco, 
Calif., allows the user to create 3-D models much like an 
artist would use a lump of clay to sculpture an object, only 
digitally. As an example, tools such as "metaballs" which are 
spheres assigned with influence parameters fuse together to 35 
form the desired object. For more detailed objects, "poly- 
gons" such as triangles or squares defining a small section of 
the object to be created may be used. Another example 
would be "splines" which are curves that define a surface 
area of an object to be created. Details of the creation of 3-D 40 
models by software may be found in George Maestri, 
"Digital Character Animation" (New Riders Publishing, 
1996). However, due to the complexity and difficulty of 
generating 3-D models with software, many have been 
deterred except those with special skills in the area. For 45 
example, the user may need to be artistic as well as tech- 
nically minded. Further, years of experience may be required 
before the techniques of 3-D modeling can be mastered. Due 
to these obstacles, the average consumer, in general, is not 
able to use the 3-D software to generate 3-D models. 50 

The generation of 3-D data is relatively slow and, as 
mentioned above, to an average consumer, time consuming 
and difficult. By using a 3-D imaging device system which 
can capture objects and subsequently be used to create 3-D 55 
data would allow consumers without special expertise to 
generate 3-D models of real objects expeditiously and with 
ease. Accordingly, the present invention is directed to a 
method and apparatus for generating 3-D data using cap- 
tured stereoscopic images with imaging devices which in 6Q 
turn is used to generate 3-D models of target objects. 

BRIEF SUMMARY OF THE INVENTION 

A method and apparatus for extracting three-dimensional 
(3-D) data from a target object is disclosed. A plurality of 65 
markers are formed on the object. A plurality of images are 
captured of the object. A first point is designated from a 
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marker from one of thejmagesj md a line equation corre^ 
sponging t otne nrst point is determined. A second point in 
a marEeTirfanother image corresponding to the first point is 
determined and a second line equation corresponding to the 
second point is determined. The intersection of the two line 
equations is determined. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The objects, features and advantages of the method and 
apparatus for the present invention will be apparent from the 
following description in which: 

FIG. 1 illustrates an exemplary three-dimensional (3-D) 
imaging device system; 

FIG. 2 illustrates another exemplary 3-D imaging device 
system; 

FIG. 3a illustrates a target object to be stereoscopically 
imaged, 

FIG. 36 illustrates a stereoscopic image of the target 
object, 

FIG. 3c illustrates extracted 3-D data, and 

FIG. 3d* illustrates surface structure formed on 3-D data; 

FIG. 4 illustrates an exemplary method of capturing 
structural light data of a target object using visible light 
source and textural data; 

FIG. 5 illustrates an exemplary method of capturing 
structural light data of a target object using non-visible fight 
source and textural data; 

FIG. 6 illustrates an exemplary calibration procedure; 

FIG. 7 illustrates an imaging device to be calibrated 
according to a chosen coordinate system; 

FIG. Ha illustrates an exemplary calibration target and 

FIGS. Sb-Hd illustrate views of the calibration target from 
the image sensor's perspective as the image sensor is being 
calibrated; 

FIG. 9 is an exemplary instrument to determine a pin hole 
length of an imaging device to be calibrated; 

FIG. 10 is another exemplary embodiment of a calibration 
target; 

FIG. 11 illustrates an exemplary 3-D data extraction 
procedure from a stereoscopic image of an object; and 

FIG. 12 illustrates another view of an exemplary 3-D data 
extraction procedure. 

DETAILED DESCRIPTION OF THE 
INVENTION 

In creating three-dimensional (3-D) models of real 
objects, a multitude of images of real objects are taken from 
different positions to exploit the differences in the objects 1 
projection. The multitude of images may be captured, for 
example, by an imaging device such as a camcorder or a 
digital camera comprising an image sensor. The image 
sensor generally comprises a photosensitive pixel array 
where each pixel captures an incident light falling on it. 
Thus, the combination of the pixels within the pixel array is 
able to capture an image from the incident light. Generally, 
a surface of the object to be re-constructed into a 3-D model 
should be present in at least two images since the 
re -construction is based on the surface intersection of the 
two images. These two or more images (i.e. stereoscopic 
image) of the object having the above characteristics are 
suitable for subsequent processing into 3-D models. 

In taking a stereoscopic image of an object, several issues 
are taken into consideration. First, the object should not 
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change its position or shape while the images are taken. ing suitable geometric structures. An example of a triangu- 

Second, the object should be such that the features on its lated structure is illustrated in FIG. 3d Textural data may be 

surface are located identically for all images. This is not true applied to the triangulated structure by using, for example, 

for specular reflections where the features change locations True Space, a software commercially available from 

as the location of the imaging device is changed. Third, the 5 Caligary, Mountain View, Calif. Generally, textural data 

path of the light should be essentially straight for the comprises material information such as physical surface 

invention to function properly. In other words, the projection properties of an object and may also comprise color infor- 

of an image should not be distorted due to modifications in mation of the object. Alternatively, the images may be stored 

its path. As an example, glass should not be in the path of the in the imaging devices 12 and 13 to be processed at a later 

projected image since glass has the tendency to bend a path 10 ^ me eliminating the need for a computing device in the 3-D 

of light. However, one skilled in the art will recognize that imaging device system. Generally, "calibration" 

an image may be compensated for the discrepancies if the information, to be described further below, relating to the 

mentioned conditions exist during stereoscopic image cap- 3-D imaging device system 10 is stored in a memory device 

ture. which may be coupled with or may be part of the computing 

FIGS. 3a-3c illustrate an example of 3-D data being 15 device 19. However, in a 3-D imaging system 10 where the 

extracted from a target object. FIG. 3a is the target object, computing device 19 is not used, the system 10 may 

in this instance, a face. By capturing images of the object comprise a memory device to store calibration information 

jrom^different^positions using an imaging device, face 3^ or the calibration information may be separate from the 

and face 34 may be captured as Ulustratedin FIG. 3b which system and introduced when the stereoscopic images are 

is a stereoscopic image of the face. One c ould verify that the 20 being converted into 3-D data. 

projection ot ine iace as captured by "the imaging device The 3-D imaging device system 10 may further comprise 

changes predictably as the position of the imaging device is a light device 16 to project an originating light beam and a 

changed. Note the images captured by the imaging device jittracting device 17 to split the beam intoZln-^Quat^ ^ 

contain only two-dimensional (2-D) data because each "patfer^ pf.linrj; grids^dotsjbrjmy omer geometrical p ajtcras._ . ^ ^ 

image is contained in the plane of the image sensor which is 25 As an example, the diffracting device may be one commer- C. 0 (, I I 

two dimensional. However, where there is an intersection of dally available from Digital Optics Corporation, Charlotte, 

at least two images of a surf ace of the face from two different N.C. or Rochester Photonics, Rochester, N.Y. The term ^ 

positions, a depth Z value may be obtained for each point in "strucUuejMight" will be understood to mean structures *ff(li£ZVA& 0 

one image that has a corresponding point in the other image cTmprSn^ines, strips, segmented lines, grids, dots, etc. #7" 

with respect to an arbitrary coordinate system having X, Y, 30 produced by a light source. The reason for the structured 

Z coordinates that is common to both image sensors. light is to provide a structure to the target object that is easily 

Furthermore, the positions of the two corresponding points recognizable by a computing device. In other words, it is 

in the two image sensors expressed by X', Y' coordinate in difficult for the computing device to match one point in one 

the image plane may be combined to produce a 3-D point image to the corresponding point in another image with 

(i.e. X, Y, Z value) which may be one of the points in 3-D 35 information obtained from the natural features of the target 

data. Thus, 3-D data is a plurality of points in 3-D space object alone. As an example, if a human face is tt%object to 

identified within a coordinate system to form a 3-D point be 3-D modeled and the eye is the feature to be matched, the 

image of the target object, an example which is illustrated in computing device may err because it may not be able to 

FIG. 3c. Obtaining X, Y, Z value of a point in the target distinguish between the two eyes in the other corresponding 

object will be described further below. Examples of 3-D 40 image or the other features. However, by using structured 

imaging device systems to capture stereoscopic images of light, the contours of the object can be easily referenced by 

objects will now be described. a computing device in terms of the location ot the vertical 

FIG. 1 illustrates an embodiment of a 3-D imaging device lines, for example. In one embodiment, each vertical line in 

( system. The 3-D imaging device system 10 illustrated is tne structured light may be coded to distinguish one vertical 

\ shown with two imaging devices which for ease of under- 45 line from another. Coding is desirable where the contour of 

\ standing the invention will be designated left imaging device a target object causes the vertical lines to emerge, for 

J 12 and right imaging device 13. The designation is men- example. In this instance, the computing device may err by 

tioned throughout the description, however, one skilled in jumping from one vertical line to another. By providing a 
/ the art would recognize from reading the description that the ' code for each fine, the computing device knowing the code 

/ designation may be interchangeable and further, the inven- 50 of the vertical line will detect an error when the code of the 
/ tion is applicable where more than two-imaging , devices are_ line being traced has changed and may be able to re-trace 

' used or under suitable conditions, where only one imaging back to correct the error. The lines may be coded by using 

/ device is used. Each imaging device 12 and 13 comprises an a suitable filter 18, for example, coupled with one end of the 
jmagejtensorJlXan d J.5Jhat, is.able Jo capture. an image"or ^ diffracting device 17. For example, the code may be the 

a target object. The 3-D imaging device system 10 may 55 segmentation of the vertical line into a pattern different from 

inciude a computing device 19 to process a stereoscopic 1 the other vertical lines in the close proximity. The filter 18 
image captured by the imaging devices 12 and 13 into 3-D. ' may have slits allowing the vertical lined structured light to 

data. The computing device may be a microprocessor, an be emitted but may have blockages in the slits corresponding 

arithmetic logic unit (ALU) or any other devices capable of to the desired segmented patterns of vertical lines, 

processing data information. In one embodiment, the com- 60 One embodiment of the light source 16 may be a laser 

puting device 19 may even process the 3-D data into 3-D , emitter. The light source 16 and the diffracting device 17 

models depending on the sophistication of the underlying may project a laser light, for example, vertical lines onto the 

software. As an example, 3-D data may be "triangulated*' target object. The description of the vertical lines should by 

(i.e. forming the surface of the object by forming triangles no means be interpreted as limiting the scope of the inven- 

with every three points of the 3-D data) using conventional 65 tion. For example, horizontal lines may be used depending 

algorithm such as Delaunav' s algorithm. O ne skilled in the on the orientation of the cameras. Furthermore, it may be a 

-art will recognize that other algorithms may be used includ- grid of dots or segmented lines, etc. FIG. 4 illustrates one 
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example of capturing a stereoscopic image of a target object 
using visible light source, and textural data. Block 41 
illustrates a struc t ured fight p roj e clj^ j on to ^ 

T)y a lig ht srmrra l7T*With thr. sln'i'"hir^' it phi prnjrrt^ 

block 42 illustrates a stereoscopic image of the object 
captured by the left 12 and right imaging devices 13 of the 
3-D imaging device system 10. Once the stereoscopic image 
of the object with a structured light (i.e. structural light data) 
is captured, block 43 illustrates the light source 16 is 
switched off. Block 44 illustrates simultaneously or as close 
to simultaneously as possible, an image is captured by one 
of the left imaging device 12 and right imaging device 13 to 
obtain the textural data. Although multiple images may be 
taken by the left 12 and ngnrimagjuag deyices_13 to obtain 
texturaldata. generally^ a smde^imafie^from ^ pne imaging 
device may be sufficient . The reason for the simultaneous- 1 
ness is to match as closely as possible the stereoscopic image 
with the structural light data to the image with textural data. 
It should, however, be noted that where the 3-D imaging 
device system 10 and the target object are relatively 
stationary, simultaneousness is no longer important. 
Furthermore, in one embodiment, the textural data may be 
independent to the stereoscopic image, for example, where 
the textural data may be computer generated using conven- 
tional methods, textural image is not required. 

In another embodiment, light source 16 may emit light in 

"^^jP^^^^^S^^.fP.j jPjj; m tnis instance, the image 
sensor 14, 15 of the imaging device 12,13 may be designed 
to enable simultaneous capture of_the jdsibleLiS s ht jextural' 
oaia and ' Jnl'ra -red structured. lightdata. T Jie. image sensor s 
jnayjje equipped tojsimultaneciuslv capture visible 
and j^ra^5^ ^^^^^^^^^S ES|^SEr 'filters'. 
"As an example, an image sensor comprising a 2x2 square 
pattern of red, green, blue (RGB) and infra-red (IR) pixels 
may be created using existing commercial Color Filter Array 
(CFA) materials, taking advantage of the fact that these 
materials are transparent to IR radiation. By a simple overlay 
of two CFA colors (e.g. R, B) that have no overlapping 
transmittance in the visible portion of the spectrum, it is 
possible to create a composite filter element which blocks 
the visible light and transmits only IR. I^teqJltej s.a re r use ■ d t 
tp^forjja.the^composite jto h as^ 

a visible radiation pass^sr^ctmm tta 
other^so^j^hat there Js_ substantially no t ransm ittance oP 
wsjblejy^^^ "filter"?o^^^45 
from the c(^bj^fS^ ^ w i 1 BSlwo~fA^sf : 'll r mbre - than" t wo 
niters are usea^lnen'eacr]^^ radiation pass 

spectrum such that the resulting composite filter is substan- 
tially opaque to visible light. This composite filter element 
is thus an IR pass filter, because each of the component 
filters used to form the composite filter is substantially 
transparent to IR. The deposition of the CFAs are accom- 
plished by photo-lithographic techniques well known to the 
semiconductor industry. Further information on RGB and IR 
image sensor may be found in a pending application titled 
^Infrared Pixel Sensor and Infrared Signal Correction", Ser. 
No. 75/041,976, filed on Mar. 13, 1998. 

FIG. 5 illustrates one example of capturing structural light 
data of a target object using non-visible light source, and 
textural data. Block 51 illustrates the non-visible light 
source projecting a structured light onto the target object. 
Block 52 illustrates a stereoscopic image taken by the left 
imaging device 12 and right imaging device 13. Block 53 
illustrates at least one of the imaging devices 12, 13 gener- 
ating color outputs (e.g. red, blue, green) for textural data 65 
and both imaging devices 12, 13 generating non-visible light 
output (e.g. infra-red) for processing of structural light data. 
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In another embodiment, the imaging device system 10 
comprises a combination of monochrome imaging devices 
such as black and white imaging devices and color imaging 
devices. Generally, where color imaging devices are used to 
capture both the texturaLdata^a noLthe^t^ data of ^_ 

thetarge ^ object, Jh 

compromised. JThis arises from color generally being 
SeBnedbym^eor more pixels (e.g. red, green and blue) that 
is also used to define a point in the feature, hence, positional 
information is spread over a group of pixels. By using one 
pixel to define one point position in the feature, higher 
resolution may be obtained. In using monochrome imaging 
devices to capture positional information of the features, a 
higher degree of accuracy may be achieved. 

FIG. 2 illustrates an embodiment of the 3-D imaging 
device system described above. The 3-D imaging device 
system 20 may comprise a pair of monochrome imaging 
devices designated here as a left monochrome imaging 
device 22 and right monochrome imaging device 23. The 
pair of monochrome imaging devices 22, 23 capture a 
stereoscopic image of a target object that comprises struc- 
tural light data of the object. The structured light is produced 
by a light source 16 and a diffracting device 17 in a manner 
described previously. Where coding ot the structured light is 
desired, an appropriate filter 18 may be coupled with the 
diffracting device 17. T he 3-D imaging device system 20 
further comprises a color imaging device 24 that captures 
the textural data of the target object. When textural data is 
to be obtained, the light source 16 is prevented trom emitting 
stractured light if the light is_in the visible spectrum. 
However, if the light is in the non-visible spectrum, the 
monochrome imaging devices 22, 23 and the color imaging 
device 24 may takE an image of the target object simulta- 
neously. In this instance the monochrome imaging devices 
should be adaptable to capture structural light data produced 
by the light in the non-visible spectrum. As an example, 
where the non-visible fight source is an infra-red fight 
emitter as described previously, the monochrome imaging 
devices may be an infra-red imaging device. It should be 
noted that all three imaging devices 22, 23, 24 should be 
calibrated with respect to a chosen coordinate system such 
that the captured positional information obtained by the 
Tlflnnr hmm ^rnfigi n Pi devices 22. 23 m ay be combined with 
the textural data of the color imag in g device 24 to form a 
3-D model with hi^h degree of resolution. Before any 



stereoscopic images are taken, all imaging devices in a 3-D 
imaging device system should be calibrated which will be 
apparent with the description below. 

Referring to FIG. 1 as an example, calibration is per- 
formed on imaging devices 12 and 13 to determine each 
position and orientation of the imaging devices before any 
stereoscopic images are captured. By performing 
calibration, the imaging devices are placed in a chosen 
coordinate system to be described further below that allows 
the computing device used to create the 3-D data to know the 
relative position of the imaging devices in the chosen 
coordinate system. With the position of the imaging devices 
known, features of the captured stereoscopic images may be 
correlated together to form a combined input in order to 
60 form the 3-D data for 3-D modeling. To illustrate this point, 
imagine two imaging devices in 3-D space taking an image 
of the same object to form a left image and a right image of 
the object which is the stereoscopic image of the object. Due 
to the differences in the two images, stereoscopic matching 
can take place. Stereoscopic matching is a process where a 
point feature in one image is matched with the correspond- 
ing point feature in the other image. While the human visual 
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system can readily detect the various features of the left image sensor may be measured as a deviation from the 

image and the right image, and correlate the two images reference orientation. In one embodiment, the deviation may 

together, a computing device performing a similar function be determined through the remaining three positional values 
would need to define the various features in terms of that correspond to Cx, Cy and roll which will be described 

coordinates in a coordinate system. The relevant information 5 with more detail further below. Note thai the orientation is 

from this activity is the set of coordinates for each image such that the image plane center is on the axis formed by the 

which determines the location of the features in the image. origin of the chosen coordinate system and the pin hole 

The coordinate set of these features in all images, together location as described above. 

with the position of the imaging devices with which each In one embodiment, a calibration target may be used to 

image was taken can then be used to determine the original 10 determine the position and orientation of the imaging device 

location in 3-D space of the identified feature. in a chosen coordinate system. For illustration purposes, the 

FIG. 6 illustrates one embodiment of performing calibra- polar coordinate system is used. An exemplary calibration 

tion. For calibration purposes, a total of at least six positional target 75 comprising a diamond represented by a dot on each 

values may be required for a complete description of the corner 76 and the center 77, totaling five dots as illustrated 

position and orientation of an imaging device with respect to 15 m FIG. 8a may be used. However, it will be apparent to one 

a chosen coordinate system. It should be noted that the skilled in the art that other configurations and shapes may be 

positional values are determined for each imaging device in used to achieve a desired result. Note that a calibration target 

the 3-D imaging device system. In one embodiment, the mav be a drawing or sets of points on a piece of paper or it 

imaging device may be defined as an image sensor 15 with may be an actual object. If an actual object is used, the object 

a pin hole P projecting from the center and normal to the 20 should have features that may be used as reference points, 

image sensor 15 at a predetermined length (i.e. pin hole As an example, a face of a cube may be used as a calibration 

length to be described further below) as illustrated in FIG. object using the corners of the face as reference points with 

7. A pin hole of an imaging device is a fictitious point in perhaps, another reference point defined at the center by the 

space located a fixed distance from and normal to the center intersection of two diagonal imaginary lines, each line 

of the image sensor where all the incident light correspond- 25 connecting two corners of the face, 

ing to an image enters the imaging device to project the Referring back to FIG. 7, the coordinate of the pin hole P 

image on the image sensor. The position of the imaging of the imaging device 13 may be defined according to a 

device may be determined by the position of the pin hole in chosen coordinate system the origin of which, for example, 

the chosen coordinate system. In one embodiment, the may be the center of the calibration target 75 and having a 

chosen coordinate system may be a cartesian coordinate 30 X-axis that may be horizontal to the calibration target, Y-axis 

system with the origin and X, Y, Z, axis designated that may be vertical to the calibration target, and Z-axis that 

arbitrarily, thus, three of the positional values may be X, Y, may be normal to the calibration target as illustrated. It 

Z, corresponding to the position of the pin hole in the chosen should noted that the chosen coordinate system should be 

coordinate system. In another embodiment, polar coordinate the same for all imaging devices to be calibrated in a 3-D 

system may be used and similarly the origin and radius, 35 imaging device system so that each imaging device would 

latitude angle, longitude angle reference are designated have a common chosen coordinate system. The coordinate 

arbitrarily, thus, the position of the pin hole may be defined of the pin hole P may be defined in Radius, Latitude angle, 

in radius, longitude angle, and latitude angle in the chosen Longitude angle, corresponding to three positional values 

coordinate system. with respect to the center dot 77 of the calibration target 75. 

The orientation of the imaging device may be determined 40 Radius, latitude angle, and longitude angle can readily be 

by the orientation of the image sensor with respect to the produced from the description of the position in the cartesian 

chosen coordinate system. In determining the orientation of coordinate system defined by the X, Y, and Z axis which is 

the image sensor, each imaging device may be designated an conventional. 

imaging device coordinate system. For example, the origin The orientation of the image sensor 15 may be determined 

of the imaging device coordinate system may be the pin hole 45 by its center 72 with respect to an axis M defined by the 

of the imaging device. The Z'-axis of the imaging device center dot 77 of the calibration target 75 and the coordinate 

coordinate system may be the axis passing through the pin of the pin hole P. The center dot 77 of the calibration target 

hole and the center of the image sensor. The X'-axis and the 75 which lies on the axis M will be imaged at the image 

Y-axis of the imaging device coordinate system may be sensor 15 representing the deviation 74 from the center 72 

parallel to a horizontal and vertical side of the image sensor 50 of the image sensor 15 with respect to the axis M. From the 

respectively. One skilled in the art will recognize that deviation 74, the center 72 of the* image sensor 15 may be 

different origin and orientation of the axis may be used for aligned with the axis M by rotating the image sensor 15 

the imaging device coordinate system. In one embodiment, about the X'-axis (Cx) and the Y'-axis (Cy) with respect to 

a polar coordinate system may be used where initially, an the pin hole P in the imaging device coordinate system 

imaging device to be calibrated may be placed in an orien- 55 described above until center 72 corresponds to the location 

tation in the chosen coordinate system where the pin hole of deviation 74. The angular values of Cx and Cy corre- 

may he at the origin of the chosen coordinate system and the sponds to two of the remaining positional values. The roll of 

center of the image sensor may he at the Z-axis of the chosen the image sensor 15 defines the rotational orientation of the 

coordinate system, the image sensor intersecting the Z-axis image sensor 15 in the Z'-axis in the imaging device 

at a distance of a pin hole length. The X'-axis and the Y'-axis 60 coordinate system. Roll is compensated for by rotating the 

of the image sensor may be parallel with the X-axis and image sensor 15 along the Z'-axis until the Y-axis of the 

Y-axis of the chosen coordinate system respectively. When coordinate is parallel with the Y'-axis of the image sensor 

the pin hole is moved in radius, longitude angle, latitude from the image sensor's perspective, for example. The 

angle to its actual position in the chosen coordinate system, angular value of the roll corresponds to the last remaining 

the image sensor would also move from its initial orientation 65 positional value. 

to a known orientation designated as reference orientation in It should be noted that the exemplary six positional values 

the chosen coordinate system. The actual orientation of the may be dictated according to the sequence in which the 
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values are applied. In other words, positional values for one 
sequence may not be interchangeable with positional values 
for another sequence. For example, positional values 
obtained for the sequence: Cx, Cy, roll, latitude, longitude, 
radius may be different from positional values obtained for 5 
the sequence: roll, Cx, Cy, longitude, latitude, radius. Hence, 
positional values are identified with the sequence of the 
positional values taken. 

Referring back to FIG. 6, it should be noted that the 
sequence illustrated should by no means be construed as a io 
limitation and one skilled in the art will recognize that other 
sequences may be used. Block 61 illustrates determining the 
pin hole length which is the length distance of the pin hole 
perpendicular and center to the image sensor. The pin hole 
length for each imaging device in the 3-D imaging device 15 
system should be determined. The pin hole length 
(sometimes known as focal point length) is generally given 
in the manufacturer's specification of the imaging device. To 
obtain a more accurate pin hole length for the individual 
imaging devices, the following instrument may be used as 20 
illustrated in FIG. 9. 

The accuracy of the pin hole length PL is important in that 
from the pin bole length, sampled points of a target object 
may be translated to a coordinate on the image sensor. For 
illustration purposes, the right imaging device 13 is used. 25 
The pin bole length PL of the imaging device 13 may be 
determined by placing the imaging device 13 on a rail 91 
which has two rectangles 93, 95 that slide back and forth 
along the axis of the rail 91. Let the two rectangles 93, 95 
be designated first rectangle 93 and second rectangle 95. 30 
Furthermore, at least one of the defining length of a rect- 
angle (commonly referred to as horizontal or vertical length) 
should be known for each rectangle. In this example, the 
horizontal half length of the second rectangle 95 is known 
which is A and the horizontal half length of the first rectangle 35 
93 is known which is B. The horizontal half length B should 
be made smaller than the horizontal half length A. The ratio 
R is then the ratio of horizontal half length B over horizontal 
half length A. Both rectangles should be mounted such that 
the center line of the rail 91 is normal to the two rectangles. 40 
Furthermore, the center of the rectangles 93, 95 should 
coincide with the center of the rail 91. The first rectangle 93 
and the second rectangle 95 should further be parallel to 
each other. First rectangle 93 must furthermore be slidable 
and at all times comply with the requirements outlined 45 
above. By sliding the first rectangle 93 in the direction of the 
imaging device 13, while the second rectangle 95 remains 
stationary, at a certain distance on the rail 91, from the image 
sensor's 15 perspective, the projection of the rectangles' 
defining lengths will coincide on the image sensor. At this 50 
point, the image of horizontal length B of the first rectangle 
93 and the image of horizontal length A of the second 
rectangle 95 passes through the pin hole P to project a same 
length X on the same location of the image sensor 15 as 
illustrated in FIG. 6. Knowing the distance between the first 55 
rectangle 93 and second rectangle 95 which is C measured 
on the rail 91, and the length of the projection on the image 
sensor measured by X, the pin hole length PL may be 
defined by the formula 

60 

Generally, the measurement inside the imaging device is 
determined in pixel units. Note that pin hole length PL 
obtains its dimension from projection X. Since projection X 65 
is measured in pixel units, pin hole length PL is also 
measured in pixel units which is adequate in practice. 
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Referring back to FIG. 6, block 62 illustrates determining 
Cx and Cy of the image sensor. FIG. Sb illustrates an image 
of the calibration target 75 captured by an image sensor 15. 
From the captured image, image sensor 15 is computation- 
ally moved so that the center 72 of the image sensor 
coincides with the imaged center 77 of the calibration target 
75. The movement in pixel units in the X'-axis and in the 
Y'-axis with respect to the imaging device coordinate cor- 
responds to the Cx and Cy respectively. Alternatively, 
because the pin hole length is known as described 
previously, Cx and Cy may be also defined in terms of the 
angular rotation in the X'-axis and Y'-axis with respect to the 
pin hole P. When the target center 77 coincides with the 
center 72 of the image sensor 15, the center of the image is 
aligned with axis M (see FIG. 7) passing through the pin 
hole and the origin of the chosen coordinate system. 
Through Cx and Cy, the orientation of the image sensor 15 
may be determined with respect to axis M and the pin hole 
P. As the calibration target 77 is relocated to the center 72 of 
the image sensor 15 from the image sensor's perspective, the 
corner dots 76 representing the calibration target are also 
computationally moved to represent the view from the 
image sensor in its new position. The relocated calibration 
target is illustrated by FIG. 8c. 

Below is an exemplary fragmented program in Visual 
Basic that illustrates the above description. 

For the example, the following definitions are required: 

VERTEX2D is describing one point of an image. Its 
relevant members are the x and y coordinate (i.e. dx, dy); 

VTXLIST2D is a list of VERTEX2D objects with asso- 
ciated housekeeping storage (i.e. calibration target where 
member 0 is center point, member 1 is top point, member 2 
is bottom point, member 3 is left point, member 4 is right 
point); 

VERTEX3D is describing one point in 3D space. Its 
relevant members are the x, y, and z coordinates (dx, dy, dz); 
and 

VTXLIST3D is a fist of VERTEX3D objects. 

Public Sub gs_UndoCoxCoyWjm(InVec As 
VTXLIST2D, res As VTXUST2D, Coflf As VERTEX2D) 
'the offsets are returned in Coff 
Dim vl As VERTEX3D 
Dim v2 As VERTEX3D 
Dim v3 As VERTEX3D 
Dim vtx2tmpl As VERTEX2D 
Dim vRot As VERTEX3D 
Dim dCosAngle As Double 
Dim dAngle As Double 
Dim i As Integer 
'figure out the x and y offsets 
Coff.dx-v2.dx 
Coff.dy-v2.dy 

'work to simulate the effect of un-doing the off-center of the 

imaging device 
'get the normal and rotation angle 
vl.dx-0 
vl.dy-0 

vl.dz-f2_dCAMF 
v2.dx-InVec.vx2A(0).dx 
v2.dy-InVec.vx2A(0).dy 
v2.dz-£2_dCAMF 

Call gs_CosAngle3(vl, v2, dCosAngle) 'get the cosine of 

angle between vl and v2 
Call gs_ArcCos(dCosAngle, dAngle) 'get the angle from 

the cosine 

Call gs_orth3(vl, v2, vRot) 'get an orthogonal vector to the 
plane spanned by vl and v2. That is the vector around 
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which the picture has to be rotated to bring the center position of the pin hole with respect to the calibration target. 

point into the center of the image plane. The latitude of the pin hole is determined by the length 

* rotate all vectors in the target vertices list to undo the Cx between the top and center dot and the length between the 

and Cy effect bottom and the center dot of the calibration target. The 

For i=0 To I nVec.lNum Vertex- 1 5 image sensor is computationally moved radially around the 

vl.dx«In Vec.vx2A(i).dx center point of the calibration target in a plane defined by the 

vl.dy=InVec.vx2A(i).dy Y-axis and the pin hole location. The orientation of the 

vl.dz^dCAMF 'dCAMF is the distance from the image imaging device maintains the condition that the center of the 

sensor to the pin hole locations (in pixels) image sensor remains on the axis defined by the center of the 

Call gs_rot3dVec(dAngle, vRot, vl, v2) 10 calibration target and the relocating pin hole. The image 

Call gs_project3Planar(£2_dCAMF, v2, v3) sensor is moved until the length between the top and center 

res.vx2A(i).dxov3.dx dot and the length between the bottom and center dot are 

res.vx2A(i).dy=v3.dy equal as viewed by the image sensor. At this point, the pin 

res.vx2A(i).bRagoTrue hole has computationally been moved into the X-Z plane 

Next i is and the angle the image sensor has computationally moved 

End Sub is the latitude. The corner dots representing the calibration 

Block 63 of FIG. 6 illustrates determining the roll of the target are computationally transformed to represent the view 

image sensor. The roll is the rotation of the image sensor from the image sensor's perspective due to the positional 

around the Z'-axis of the imaging device coordinate system, change of the image sensor. 

in this instance, the Z'-axis corresponding to the axis M 20 Below is an exemplary program in Visual Basic that 

passing through the pin hole P and the center 72 of the image illustrates the description above: 

sensor 15. Referring to FIG. 8c, once the image sensor is Public Sub gs_UndoLatitudeWjm(src As VTXUST2D, res 

relocated so that the image sensor center 72 coincides with As VTXLIST3D, proj As VTXLIST2D, dLatitude As 

target center 77, the projection of the Y-axis of the target 75 Double) 

onto the image sensor is compared with a Y'-axis passing 25 'Find out the latitude through comparison of the angles from 

through the center and parallel to vertical sides of the image midpoint in both 

sensor 15 from the image sensor's perspective. The angle 'directions of y 

deviation between the Y-axis and the Y'-axis is the roll of the Dim dml As Double 

image sensor. The roll is compensated for by computation- Dim dm2 As Double 

ally rotating the image sensor along the axis M until the 30 Dim dm3 As Double 

Y-axis of the calibration target is parallel with the Y'-axis of Dim vl As VERTEX3D 

the image sensor. As the roll is compensated, the corner dots Dim v2 As VERTEX3D 

representing the calibration target are also computationally Dim v3 As VERTEX3D 

moved to represent the view from the image sensor's Dim v4 As VERTEX3D 

perspective due to the roll compensation. The roll compen- 35 Dim i As Integer 

sated calibration target 75 is illustrated by FIG. Hd. Dim dAlpha As Double 

Below is an exemplary program in Visual Basic that dml«src.vx2A(l).dy/f2_dCAMF 

illustrates the description above: dm2«src.vx2A(2).dy/f2_dCAMF 

Public Sub gs_UndoRollWjm(src As VTXLIST2D, res As If Abs(dml+dm2)>a000000000001 Then 

VTXLIST2D, dTwist As Double) 'dTwist is the detected 40 dm3=2*dml*dm2/(dml+dm2) 

roll angle' dAlpha=Atn(dm3)-const_PI/2 

'undo the roll after the imaging device orientation has been Else 

corrected for Cx,Cy dm3=lE+100 

Dim dalphal As Double dAlpha-0 

Dim dalpha2 As Double 45 End If 

Dim dAlpha As Double 'range of dalpha is -90 to +90 deg 

Dim vl As VERTEX3D If dAlpha<-const__PI/2 Then 

Dim v2 As VERTEX3D dAlpha-dAlpha+const_PI 

Dim i As Integer End If 

'rotation around the z axis with angle defined by atn x/y 50 dLatitude-dAlpha*190/const_PI 

dalphal-Atn(src.vx2A(l).dx/src.vx2A(l).dy) Dim vpLatVec As VERTEX3D 

dalpha2-Atn(src.vx2A(2).dx/src.vx2A(2).dy) Dim vpl As VERTEX3D 

dAlpha-(dalphal+dalpha2)/2 'take arithmetic mean Dim vp2 As VERTEX3D 

dTwist— dAlpha* 190/const_PI Dim vp3 As VERTEX3D 

'simulate undoing the roll on the five calibration points of 55 Dim vp4 As VERTEX3D 

the image Dim vp5 As VERTEX3D 

For i-LBound(src.vx2A) To UBound(src.vx2A) Dim vll As VERTEX3D 

vl.dx-src.vx2A(i).dx Dim vl2 As VERTEX3D 

vl.dy-src.vx2A(i).dy Dim vPt As VERTEX3D 

vldz-0 60 'correct the display: 

Call gs_rot3dZ(dAlpha, vl, v2) 'create a vector which is tilted into the direction of the 

res.vx2A(i).dx-v2.dx latitude 

res.vx2A(i).dy-v2.dy vpLatVec.dx-0 

Next i vpLatVec.dy-Cos(dAlpha) 

End Sub 65 vpLatVec.dz-Sin(dAlpha) 

Block 64 of FIG. 6 illustrates determining the latitude vpl.dx-0 

position of the pin hole. Latitude determines the vertical vpl.dy-0 
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vpl.dz-0 

Call gs_vec3ToNormalPlane(vpLatVec, vp2, vp3) 

vp4.dx«l 

vp4.dy=0 

vp4.dz=0 5 

vp5.dx=vpLatVec.dx 

vp5.dy=vpLatVec.dy 

vp5.dz=vpLatVec.dz 

'shift the plane from the pin hole to the center of the ccd 

vpl.dz«vpl.dz-£2_dCAMF 10 

vp2.dz«vp2.dz-£2_dCAMF 

vp3.dz=vp3.dz-£2_dCAMF 

vp4.dz=vp4.dz-f2_dCAMF 

vp5.dz=vp5.dz-£2_dCAMF 

vll.dx=0 15 

vll.dy=0 

vll.dz«0 

res.vx3A(0).dx«src.vx2A(0).dx 
res.vx3A(0).dy«src.vx2A(0).dy 

res.vx3A(0).dz=-£2_dCAMF 20 

* simulate un-doing the latitude 

For i=l To 4 

vl2.dx«src.vx2A(i).dx 

vl2.dy=src.vx2A(i).dy 

vl2.dz— £2_dCAMF 25 
If i<3 Then 

Call gt_PlaneLinelntersection(vpl, vp4, vp5, vll, vl2, vPt) 
Else 

Call gf_bPlaneLineIntersecuon(vpl, vp2, vp3, vll, vl2, 

vPt) 30 
End If 

'rotate around the x axis 

vPt.dz=vPt.dz+f2_dCAMF 

Call gs_rot3dX(^dAlpha, vPt, v3) 

'shift everything back by the f distance 35 

v3.dz=v3.dz-£2_dCAMF 

res.vx3A(i)=v3 

'project into the image sensor plane 

Call gs_project3Planar(-£2_dCAMF, v3, v4) 

proj.vx2A(i).dx=v4.dx 40 

proj .vx2A(i).dy=v4.dy 

Next i 

End Sub 

Block 65 of FIG. 6 illustrates determining longitude 
position of the pin hole. Longitude determines the position 45 
of the pin hole in the X-axis with respect to the chosen 
coordinate system. Longitude is determined by the length 
between the left dot and the center dot and the length 
between the right dot and the center dot of the calibration 
target as imaged in the image sensor. The image sensor is 50 
computationally moved radially around the calibration tar- 
get in the X-Z plane. The orientation of the imaging device 
is changed so that the center of the image sensor remains on 
the axis defined by the center of the calibration target and the 
relocating pin hole. The image sensor is moved until the 55 
length between the left dot and the center dot is equal to the 
length between the right dot and the center dot as viewed by 
the image sensor. At this point, the angle the pin hole has 
moved is the longitude. The corner dots representing the 
calibration target are computationally transformed to repre- 60 
sent the view from the image sensor's perspective due to the 
positional change of the image sensor. 

Below is an exemplary program in Visual Basic that 
illustrates the description above: 

Public Sub gs__UndoLongitudeWjm(src As VTXLIST3D, 65 
res As VTXLIST3D, proj As VTXLIST2D, dLongitude 
As Double) 



'Find out the longitude through comparison of the angles 

from midpoint in both 
'directions of x 
Dim dml As Double 
Dim dm2 As Double 
Dim dm3 As Double 
Dim vl As VERTEX3D 
Dim v2 As VERTEX3D 
Dim v3 As VERTEX3D 
Dim v4 As VERTEX3D 
Dim i As Integer 
Dim dA As Double 
Dim dAJpha As Double 

'first get the projection of point 3 into the yz plane 

Call gs__project3Planar(-f2_dCAMF, src.vx3A(3), vl) 

Call gs_project3Planar(-f2_dCAMF > src.vx3A(4), v2) 

'next find out what angle we have from 0 point to point 3 

'in the xz plane 

dml-vl.dx/vl.dz 

dm2«v2.dx/v2.dz 

If Abs(dml+dm2)>0.000001 Then 

dm3«2Mml*dm2/(dml+dm2) 

Else 

dm3= 1000000 
End If 

dAlpba«const_PI/2-Atn(dm3) 

If dAlpha>const__PI/2 Then dAlpha-dAlpha-const_PI/2 

dLongitude=dAlpha* 190/const_PI 

'simulate undoing of longitude 

For i=0 To 4 

v2=src.vx3A(i) 

'simulate shift into the pin hole plane, by making z 0 

v2.dz«v2 .dz+£2_dCAMF 

'rotate around the y axis 

Call gs__rot3dY(dAlpha, v2, v3) 

'shift everything back by the f distance 

v3.dz=v3.dz-£2_dCAMF 

res.vx3A(i)»v3 

'project into the image sensor plane 

Call gs_4>roject3Planar(-f2_dCAMF, v3, v4) 

proj.vx2A(i).dx=v4.dx 

proj.vx2A(i).dy«v4.dy 

Next i 

End Sub 

Block 66 of FIG. 6 illustrates determining the radius of 
the pin hole. Radius is the distance between the pin hole and 
the origin of the chosen coordinate system. Radius may be v 
determined, for example, in the following manner. From the 
calibration target, the distance between the corner dot and 
the center dot of the calibration target is known. From the 
image sensor, the imaged corner dot and the center dot of the 
calibration target may be measured in pixels. Because the 
pin hole length is known, the radius may be defined by the 
formula 

Radius-PL*(A'/X*) 

where A* is the distance between a comer dot and a center 
dot of the calibration target and X' is the imaged distance of 
the corner dot and the center dot at the image sensor. 

Below is an exemplary program in Visual Basic that 
illustrates the description above: 

Public Sub gs_DetermineDistanceWjm(src As 
VTXLIST3D, res As VTXLIST3D, proj As VTXUST2D, 
dDistance As Double) 

res-src 

End Sub 

In the instances where the change in the lengths corre- 
sponding to the relative dots in the calibration target 75 are 
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relatively minor, it is desirable to install an additional interpreted as limiting the scope of the invention. For 

calibration dot that is protruding from the plane where the example, horizontal lines may be used depending on the 

calibration target 75 is located to be closer to the image orientation of the imaging devices 12 and 13. Furthermore, 

sensor with respect to the remaining target dots 76, 77. depending on the sophistication of the diffracting device 17, 

Because the additional dot is closer to the image sensor, the 5 a grid of uniform dots may be projected onto the object, 

dot is more susceptible to the change in the direction of the Usinfl structured light as an example to further the under- 

image sensor than the dots 76, 77 on the calibration target standing of the invention, the light source 16 with the 

75. FIG. 10 illustrates an exemplary calibration target 101 diffracting device 17 project a structured light, in this 

having a diamond shape with a dot 108 protruding normally example vertical lines onto the target object. With the 

from the plane of the calibration target 101. This is achieved to ?"™tured light projected, a stereoscopic image of the object 

by having a triangular plane protruding from the plane " ^Vn ^f^^iftST 

where the calibration target 101 1 located as illustrated L The d ™<* J 3 ° f ' he 3_D ™ agm S ^ 10 J 10 ' U 1 

, Ct . . 6 . ACO . ... . . a flowchart of an embodiment illustrating the extraction of 

angle of the protrusion may be 45°. A calibration dot 108 is 3 . D data from a stereoscopic ^ of J object comprising 

placed on the protruding Wangle m a manner such that the a left ^ and a ^ £ B * lock m aWates deter- 

dot 108 aligns with the left dot 102 and right dot 104 of the is mining the number of vertical ^ detected by the right 

calibration target 101. image sensor corresponding to the vertical lines projected 

Referring back to the determination of the latitude of onto the target object using strucnir^ Ug hj Blnrk 114 

block 64 in FIG. 6, the image sensor having a latitude with illustrates determining the number of vertical lines detected 

respect to the calibration target 101 will detect the protrud- by the left image sensor also corresponding to the vertical 

ing dot 108 to be either above or below an X-axis passing 20 lines projected onto the target object. Block 116 illustrates 

through the left dot 102 and right dot 104 of the calibration matching the vertical lines detected by _ the right image 

target 101. The image sensor is computationally moved sensor with the vertical lines detected by the left image 

vertically until the dot 108 aligns with the left dot 102 and sensor in a correlating manner. Block 1 18 illustrates setting 

the right dot 104. At this point, the angular distance the pin a first counter at 1 reflecting the first correlating vertical line 

hole has moved corresponds to the latitude. Turning to the 25 detected at the right image sensor. Block 120 illustrates 

determination of longitude of block 65 in FIG. 6, the image converting the correlating vertical line into a plurality of 

sensor having a longitude with respect to the calibration dots. Typically, the distance between the dots is measured in 

target 101 will detect the protruding dot 108 to be at a pixel units, for example, a dot from the vertical line may be 

distance from the calibration target 101. As an example, the formed per every five pixels. Note that by controlling the 

ratio corresponding to the distance between the protruding 30 number of correlating vertical lines and the number of dots 

dot 108 and the right dot 104 in the plane where the to be converted from each vertical line, a desired number of 

calibration target 101 is located and the distance between the points in the 3-D data may be obtained. Block 122 illustrates 

right dot 104 and the left dot 102 can be predetermined, thus* determining the number of dots converted from the vertical 

the ratio is computationally calculated as the image sensor is line and setting a second counter at 1 reflecting the desig- 

computationally moved horizontally until the predetermined 35 nated dot to be the first dot. Block 124 illustrates computing 

ratio is reached. At this point, the angular distance the pin a "line of sight" originating from the designated dot and 

hole has moved corresponds to the longitude. passing through the pin hole of the right imaging device to 

When the six positional values are known for each image project the line in 3-D space. At a certain point in the line of 

sensor of the imaging devices in the 3-D imaging device sight, the coordinate unknown, the point will intersect with 

system, the calibration is complete. The computing device 40 a vertical line in the target object that correlates with the 

used to create 3-D data is able to know the relative position vertical line from which the designated dot originated. The 

of the image sensors in the chosen coordinate system by intersecting point at the target object will also correspond to 

reverse sequencing the positional values obtained. For the position of the designated dot in the right image sensor, 

example, if the sequence taken is Cx, Cy, roll, latitude, In one embodiment, the line of sight for the right image 

longitude, radius, then the computing device by reverse 45 sensor may be produced in the following manner, 

sequencing, that is computing in the sequence, radius, Referring to FIG. 7, the position of the pin hole P of the 

longitude, latitude, roll, Cy, Cx, knows the position of the right imaging device 15 is known in the chosen coordinate 

image sensor. system from the calibration of the right imaging device. 

With the calibration performed on each of the imaging Note that a pin hole P is positioned at a fixed distance PL 

devices in the 3-D imaging device system and before 50 from the center and normal to the image sensor. In this 

stereoscopic images are taken by the imaging devices 12 and example, let the image sensor 15 comprise a 640x480 pixel 

13 as illustrated in FIG. 1, for example, initially, markers array. However, one skilled in the art will recognize that 

should be placed on the target object 20 that outline the other array sizes may also be used. Using pin hole P as the 

contours or features of the object as illustrated in FIG. 3a. origin of an imaging device coordinate system, every point 

For example, vertical lines 21 may be evenly placed on 55 in the image sensor 15 may be referenced from the pin hole 

object 20. However, it should be noted that more lines 21 P in pixel values (AX", AY', PL) where AX' is the deviation 

may be added to object 20 in areas where there are fine in the X'-axis of the position of the designated dot from the 

features such as the eyes or the nose, for example. The center of the image sensor 15, AY' is the deviation in the 

vertical lines 21 may be painted onto the object 20 using Y'-axis of the position of the designated dot from the center 

fluorescent paint that is visible only in the dark such that 60 of the image sensor 15, PL is a known fixed value which is 

structural data may be obtained in the dark. Alternatively, the a fixed distance in the Z'-axis, all three values measured in 

paint used may be invisible in the visible spectrum but pixel units. Because the pin hole coordinate is known in the 

visible to radiation outside the visible spectrum such as chosen coordinate system using this coordinate and values 

infra-red or ultraviolet light. Alternatively th e vertical lines (AX', AY', PL) corresponding to the designated dot, an 



21 may be p rnj^te^ pntn the-ofaiect 20 using li^ht spurceJL^ equation corresponding to the line of sight for the designated 
sucn as infra- red laser or visible laser It w ill be appreciated dot may be produced. FIG. 12 illustrates a fine of sight LI 
ThaTihe descripdon ot vertical lines should by no means be representing the designated dot of the right image sensor 15. 
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Block 126 illustrates determining a point in the oorrelat- If gf_bLineDistance3D(ln3A, ln3B, vx3A, dDist)=False 

ing vertical line of the left image sensor 14 that corresponds Then 'result in vx3A 

to the designated dot of the right image sensor 15. Once the 'The fifth procedure above illustrates finding the intersection 

corresponding point is determined, because the coordinate of the right and left line of sights, wherein if the intersection 

of the left pin hole is known in the chosen coordinate system 5 does not occur, the point where the right line of sight is 

through calibration, and the positional value of the point closest to the left line of sight is designated as the intersec- 

may be determined in the manner described immediately tion point' 

above, an equation representing the light of sight of the The subroutine below illustrates the first procedure in further 

corresponding point may be obtained. FIG. 12 illustrates a details: 

line of sight L2 representing the corresponding point of the 10 Public Sub tc_MakeLineOfSight(vx21n As VERTEX2D, 

left image sensor 14. In one embodiment, the corresponding ln30ut As LINE3D, obj2D As Object) 

point in the left image 14 may be determined by "tracing" Dim vx2A As VERTEX2D 

the line of sight LI of the right image sensor 15 onto the left Dim dDist As Double 

image sensor 14. Because the correlating vertical lines in the dDist=50 

left and right image sensor corresponds to the same vertical 15 vx2A.dx-vx2In.dx-obj2D.f2_iCAMCx/2 'obj2D 'deter- 

line in the target object, the point of intersection of the line mines the center X'-axis of the image sensor' 

of sight from me right image sensor and the vertical line in vx2A.dy«vx2In.dy-obj2D.f2_iCAMCy/2 'determines the 

the target object as viewed by the left image sensor would center Y'-axis of the image sensor' 

be the corresponding point of the designated dot to the right 'Above two lines illustrate determining the center of the 

image sensor. An analogy will be used to explain block 126 20 right image sensor and designating as coordinate (0, 0) in 

to aid in the understanding of the procedure. ✓ the right imaging device coordinate system' 

Assuming that the designated dot Dl in the vertical line ln30ut.vx3A.dx-obj2D.f2_dLOSStart*vx2A.dx/ 

imaged at the right image sensor 15 is able to emit a beam obj2D.f2__dCAMF 

of light LI that passes through right pin hole PR of the ln30ut.vx3A.dy=obj2D.f2_dLOSStart*vx2A.dy/ 

imaging device 13 and through space. The left imaging 25 obj2D.f2__dCAMF 

device 12 would be able to detect this beam of light LI on ln30ut.vx3A.dZ~- obj2D.£2_dLOSStart 

its image sensor 14 from its field of view. The intersection 'Above three lines illustrate designating a starting point for 

of the detected beam of light LI and the vertical line in the the right line of sight, the coordinate defined in the right 

left image sensor 14 correlating with the vertical line in the imaging device coordinate system* 

right image sensor 15 from which the designated dot origi- 30 ln30ut.vx3B.dx»obj2D.f2_dLOSEnd*vx2A.dx/obj2D.£2_ 

nated may be determined. The intersection point D2 would dCAMF 

be the corresponding point of the designated dot Dl in the ln30ut.vx3B.dy=obj2D.£2_dLOSEnd*vx2A.dy/obj2D.£2_ 

left image sensor. From the intersection point D2, a beam of dCAMF 

light L2 may be projected that passes through the left pin ln30ut.vx3B.dZ«-obj2D.f2_dLOSEnd 

hole PL of the left imaging sensor which should intersect 35 'Above three lines illustrate designating an ending point for 

with the beam of light LI. The point of intersection of the the right line of sight, the coordinate defined in the right 

two beams of light LI and L2 will be the X, Y, Z, coordinate imaging device coordinate system' 

in the chosen coordinate system of a point in the target object 'Note the starting point and the ending point may be user 

corresponding to the designated dot in the right image sensor defined so that the distance between the starting point and 

which is one point of 3-D data of the target object. Of course, 40 the ending point is sufficient to intersect the target object. 

in reality, the above description is performed mathemati- As an example if the target object is 4 feet from the right 

ca My- image sensor, the starting point may be designated at 0.5 

Below is an exemplary program written in Visual Basic to feet and the ending point may be designated at 6 feet' 

be used to illustrate a mathematical procedure performing Call gs_XformCameraTo\Vorld(ln30ut.vx3A, 

the above description. 45 ln30ut.vx3A, obj2D) 

tc_MakeLineOfSight vx2A, ln3A, objA Call gs XformCameraToWorld(ln30ut.vx3B, ln30ut.vx3B, 

'The first procedure above illustrates producing a right line obj2D) 

of sight from the right image sensor where the coordinates 'Transforming the coordinates of the starting point and the 

of the right line of sight corresponds to the chosen coordi- ending point from coordinates in the right imaging device 

nale system from calibration' 50 coordinate system to coordinates in the chosen coordinate 

tc__Ln30bjMakeLn2 ln3A, ln2A, objB system' 

'The second procedure above illustrates converting the right End Sub 

line of sight from the right image sensor in the chosen 'The subroutine below illustrates converting points in the 

coordinate system to a left imaging device coordinate sys- imaging device coordinate system to coordinates in the 

tem. The right line of sight is traced onto the left image 55 chosen coordinate system* 

sensor* Public Sub gs_XformCameraToWorldWjm(vx3In As 

If gf_bCbntourLineIntersection(ln2A, cn2B, vx2B) VERTEX3D, vx30ut As VERTEX3D, obj2D As Object) 

'The third procedure above illustrates finding the intersec- Dim vl As VERTEX3D 

tion of the traced right line of sight on the left image sensor Dim v2 As VERTEX3D 

and the vertical line imaged at the left image sensor corre- 60 Dim vRot As VERTEX3D 

lating with the vertical line on the right image sensor from Dim dCosAngle As Double 

which the right line of sight originated* Dim dAngle As Double 

Call tc_MakeLineOfSigbt(Vx2B, ln3B, objB) Dim dlwist As Double 

'The fourth procedure above illustrates producing a left line Dim dLongitude As Double 

of sight from the left image sensor from the intersecting 65 Dim dLatitude As Double 

point and the coordinate of the left pin hole of the left Dim dDistance As Double 

imaging device in the chosen coordinate system 1 Call gs_rot3dCOxCOyWjm(vx3In, obj2D, False, v2) 
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vl«v2 

'The call routine above compensates for the orientation Cx 
and Cy values of the image sensor to transform the right 
imaging device coordinate system to the choseo coordinate 
system. The compensation of the Cy value which may be 
obtained by an angular rotation in the X-axis which changes 
the position of the Y'-Z' plane. The starling point and the 
ending point of the right line of sight is compensated by the 
angular rotation amount for the change in the Y'-Z' plane. 
The compensation of the Cx value may be obtained by the 
angular rotation in the Y-axis which changes the position of 
the X f -Z' plane. The starting point and the ending point of the 
right line of sight is compensated by the angular rotation 
amount for the change in the Y'-Z' plane. When the orien- 
tation Cx and Cy is compensated for the right imaging 
device, the center of the right imaging sensor is aligned with 
the pin hole of the right imaging device and the origin of the 
chosen coordinate system. The new position of the starting 
point and the ending point of the right line of sight reflects 
the perception of the points from the changed position of the 
right image sensor.' 

d1\vist=obj2D.f2_dCAMTtwist*const_PI/190 

Call gs^otia^-dlwist, vl, v2) 

vl«v2 

'The call routine above compensates for the roll of the right 
image sensor with respect of the chosen coordinate system. 
The compensation of the roll value which may be obtained 
by an angular rotation in the Z'-axis which changes the 
position of the X'-Y* plane. The starting point and the ending 
point of the right line of sight is compensated by the angular 
rotation amount for the change in the X'-Y 1 plane. The new 
position of the starting point and the ending point of the right 
line of sight reflects the perception of the points from the 
changed position of the right image sensor.' 
'move by f along z axis to move us into the center of the 
world 

dDistance«=obj2D.f2_dCAMZ 
vl.dz=vl.dZ+dDistance 

'Above three lines illustrate compensating for the radius of 
the transformed right imaging device coordinate system. 
Note that once Cx, Cy and roll has been compensated, the 
right imaging device coordinate system is aligned with the 
chosen coordinate system having the origin at the pin hole 
of the right imaging device. By moving the origin at the pin 
hole to the origin of the chosen coordinate system, the 
position of the starting point and the ending point of the right 
line of sight reflects the perception of the points with the 
right image sensor at the origin. This is performed by 
compensating for the radius, latitude angle and longitude 
angle of the right imaging device coordinate system to the 
coordinate system into the chosen coordinate system.' 
dLatitude-obj2D.a_dCAMTlat*const_PI/190 
Call gs_rot3dX(^dLatitude, vl, v2) 
vl-v2 

'Above three lines illustrate compensating for the latitude 
angle.' 

dLongitude-obj2D.f2_dCAMTlong*const_PI/190 

Call gs_rot3dY(dLongitude, vl, v2) 

vl-v2 

'Above three lines illustrate compensating for the longitude 

angle.* 
vx30ut-v2 
End Sub 

Below are exemplary codes for the subroutine call to com- 
pensate for the orientation Cx and Cy values of the image 
sensor to transform the right imaging device coordinate 
system to the chosen coordinate system/ 
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Public Sub gs__rot3dCOxCOyWjm(vx31n As VERTEX3D, 
obj2D As Object, forward As Boolean, vx30ut As 
VERTEX3D) 

Dim vx2vl As VERTEX2D 
s Dim vx2v2 As VERTEX2D 

Dim vx3vl As VERTEX3D 

Dim vx3v2 As VERTEX3D 

Dim vx3Rot As VERTEX3D 

Dim d Cos Angle As Double 
10 Dim d Angle As Double 

'create the corrected 2d coordinates 

vx2vl.dx=obj2D.Q_iCAMCOx-obj2D.£2_iCAMCx/2 

vx2vl.dy-obj2D.Q_iCAMCOy-obj2D.f2_iCAMCy/2 

'undo cox,coy through a rotation around the normal which 
15 is spanned by pinhole,cox,coy, and pinhole,0,0 ((z,x,y) as 
unit vectors)' 

'build 3d vectors for the two known points 

vx3vl.dx=0 

vx3vl.dy«0 
20 vx3vl.dZ=obj2D.£2_dCAMF 

vx3v2.dx=vx2v2.dx 

vx3v2.dy«vx2v2.dy 

vx3v2.dZ«obj2D.£2_dCAMF 

'get the rotation angle and the normal vector 
25 Call gs_CosAngle3(vx3vl, vx3v2, dCosAngle) 

Call gs_j\rcCos(dCosAngle, dAngle) 

If bForward-False Then 

dAngle = -dAngle 

End If 

30 Call gs_orth3(vx3vl, vx3v2, vx3Rot) 

Call gs__rot3dVec(dAngle, vx3Rot, vx31n, vx30ut) 
End Sub 

'The subroutine below illustrates the second procedure in 
further details.' Public Sub tc_Ln30bjMakeLn2(ln3A As 
35 LINE3D, ln2A As LINE2D, obj2D As Object) 

Dim vx2A As VERTEX2D 

Dim vx2B As VERTEX2D 

Dim vx3AWorld As VERTEX3D 

Dim vx3ACam As VERTEX3D 
40 Dim vx3BWorld As VERTEX3D 

Dim vx3BCam As VERTEX3D 

'transform the 3D line into camera coordinates 

vx3AWorld=ln3A.vx3A 

vx3BWorld-ln3A.vx3B 
45 Call gs_XformWorldToCamera(vx3AWorld, vx3ACam, 
vx2A, obj2D) 

The call routine above transforms the starting point of the 
right line of sight in the chosen coordinate system into a 
coordinate in the left image sensor plane in the left 
50 imaging device coordinate system.' 

Call gs_XformWorldToCamera(vx3BWorld, vx3BCam, 
vx2B, obj2D) 

'The call routine above transforms the ending point of the 
right line of sight in the chosen coordinate system into a 
55 coordinate in the left image sensor plane in the left 
imaging device coordinate system.' 

ln2A.vx2A-vx2A 

ln2A.vx2B-vx2B 

End Sub 

60 'Below is an exemplary call routine to transform a point in 
the chosen coordinate system to a point in the imaging 
device coordinate system. The routine below may be 
applied to the starting point and the ending point of the 
right line of sight.' 

65 Public Sub gs_XformWorldToCameraWjm(vx3In As 
VERTEX3D, vx30ut As VERTEX3D, obj2D As Object) 
Dim dAlpha As Double 
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Dim dBeta As Double 
Dim dF As Double 
Dim dDistanee As Double 
Dim dLongiiude As Double 
Dim dLatitude As Double 
Dim dlwist As Double 
Dim vx3Rot As VERTEX3D 
Dim iCOx As Integer 
Dim iCOy As Integer 
Dim iCx As Integer 
Dim iCy As Integer 
Dim vx3vl As VERTEX3D 
Dim vx3v2 As VERTEX3D 

dLongitude=obj2D.Q_dCAMTlong*3.1415926/180 
dLatitude«-obj2D.£2_dCAMTlat <, 3.1415926/180 
dTwist«obj2D.G_dCAMTrwist*3.1415926/180 
dF=obj2D.£2_dCAMF 
dDistance-obj2D.f2_dCAMZ 
Call gs_rot3dY(-dLongitude J vx3In, vx3v2) 
The above call routine applies a longitude angle of the pin 20 

hole of the left imaging device to a point in the chosen 

coordinate system.' 
vx3vl»vx3v2 

Call gs_rot3dX(dLatitude, vx3vl, vx3v2) 



through the corresponding point D2 and the left pin hole PL 
is derived. Block 128 illustrates the computing device cal- 
culating the X, Y, Z coordinate of a point on the object 
corresponding to the designated point at the right image 
5 sensor using the equations derived from block 124 and block 
126. Note that the equations have variables in terms of the 
pin hole coordinate which is defined in terms of a coordinate 
in the chosen coordinate system. Thus, the X, Y, Z coordi- 
nate of the point on the object would be a coordinate in the 
to chosen coordinate system. Block 130 illustrates determining 
if there are any more points on the vertical line to be 
processed and if so, the counter is incremented and the steps 
illustrated in blocks 120-130 executed. Block 132 illustrates 
determining if there are any more vertical lines to be 
processed. Blocks 120-132 illustrates if there are more 
vertical lines to be processed, the counter in incremented and 
the process is repeated. Otherwise the extraction of the 3-D 
data is complete and the process halts. 

The exemplary embodiments described herein are pro- 
vided merely to illustrate the principles of the invention and 
should not be construed as limiting the scope of the inven- 
tion. Rather, the principles of the invention may be applied 
to a wide range of systems to achieve the advantages 
described herein and to achieve other advantages or to 
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'The above call routine applies a latitude angle of the pin 25 satisfy other objectives as well, 



hole of the left imaging device to the point in the chosen 
coordinate system.' 
vx3vl«vx3v2 

vx3vl.dZ=vx3vl .dZ-dDistance 

'The above call routine applies a radius of the pin hole of the 
left imaging device to the point in the chosen coordinate 
system.' 

Call gs_rot3dZ(dTwist, vx3vl, vx3v2) 

'The above call routine applies a roll of the left image sensor 

To the point in the chosen coordinate system.' 
vx3vl*vx3v2 
'apply c0x,c0y 

Call gs_rot3dC0xC0y\Vjm(vx3vl, obj2D, True, vx3v2) 
' The above call routine applies a Cx and Cv of the image 

sensor to the point in the chosen coordinate system/ 
vx3vl»vx3v2 
vx30ut«vx3v2 
End Sub 

Note that once the coordinates of the starting point and the 
ending point in the right imaging device coordinate system 
have been transformed to coordinates in the left ima ging 
'device coordinate system, the pin hole of the left imaging* 
"device, whicn may be tbe origin of the left imaging device 
coordinate system may be used to project the transformed 
starting point and the ending point onto the left image 
sensor. Stated differently, from the known coordinates of the 
starting point and the ending point in the left imaging device 
coordinate system, the known coordinate of the left pin hole, 
two equations may be determined that passes through the pin 
hole and image sensor, the two equations corresponding to 
line of sight of the starting point and the ending point. From 
the known pin hole length of the left imaging device, ( the two 
points on the location of the left image sensor may be 
determined. From the two located points, a line may be 
traced on the left image sensor corresponding to the line of 
sight of the first imaging device perceived by the left image 
sensor. 

At block 126, the computing device determines the X, Y, 
Z coordinates of the corresponding point D2 (see FIG. 12) 
at the left image sensor 14 and using the known X, Y, Z 
coordinate of the left pin hole PL determined through 
calibration, an equation representing a line L2 passing 
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What is claimed is: 

1. A method of extracting three dimensional (3-D) data 
from a target object comprising: 

performing calibration on a first imaging device and a 
second imaging device; 

compensating a starting point and an ending point coor- 
dinate; 

forming a plurality of markers on said target object; 
capturing a plurality of images of said target object; 
designating a first point from a first marker in a first 
image; 

determining a first line equation corresponding to said 
first point; 

determining a second point corresponding to said first 
point in a second marker in a second image; 

determining a second line equation corresponding to said 
second point; determining an intersection of said first 
line equation and said second line equation. 

2. The method according claim 1, wherein forming a 
plurality of markers further comprises projecting a struc- 
tured light onto said target object. 

3. The method according to claim 2, where said structured 
fight is one of lines, grids, dots, segmented lines produced by 
a light source. 

4. The method according to claim 3 wherein said light 
source is a laser emitter. 

5. The method according to claim 4 further comprising 
coding said plurality of markers such that a marker and a 
successive marker have a different coding. 

6. The method according to claim 1 wherein capturing a 
plurality of images of said target object further comprises: 

capturing a first image having a first set of markers from 

said plurality of markers; 
capturing a second image having a second set of markers 

from said plurality of markers where said second set of 

markers having at least one marker correlating with a 

marker in said first set of markers; 
determining a number of said correlating markers. 

7. The method according to claim 6, wherein said plurality 
of markers are a plurality of lines, further comprising: 



06/09/2004, EAST Version: 1.4.1 



6,125,197 



23 



24 



forming a plurality of points from said plurality of lines on 
one of said first and second images. 

8. The method according to claim 1, wherein determining 
said first line equation further comprises: 

determining a position of a first pin hole corresponding to 5 
said first imaging device capturing said first image; 

determining a relative position of said first point with 
respect to said first pin bole; 

determining said first line equation originating from said 1Q 
first point and passing through said first pin hole. 

9. The method according to claim 8, wherein said position 
of said first pin hole is respective to a first imaging device 
coordinate system, the method further comprising: 

selecting a starting point of said first line equation having 15 
a starting point coordinate in said first imaging device 
coordinate system; 

selecting an ending point of said first line equation having 
a ending point coordinate in said first imaging device 
coordinate system; 20 

transforming said starting point coordinate from said first 
imaging device coordinate system to a starting point 
coordinate in a chosen coordinate system; 

transforming said ending point coordinate from said first 
imaging device coordinate system to an ending point 25 
coordinate in said chosen coordinate system. 

10. The method according to claim 9, wherein transform- 
ing said starting point coordinate and transforming said 
ending point coordinate in said first imaging device coordi- 
nate system to said chosen coordinate system comprises: 30 

compensating said starting point coordinate and said 
ending point coordinate for an orientation Cx obtained 
from calibration of said first imaging device; 

compensating said starting point coordinate and said 35 
ending point coordinate for an orientation Cy obtained 
from calibration of said first imaging device; 

compensating said starting point coordinate and said 
ending point coordinate for a roll obtained from cali- 
bration of said first imaging device; 

compensating said starting point coordinate and said 
ending point coordinate for a latitude angle obtained 
from calibration of said first imaging device; 

compensating said starting point coordinate and said 
ending point coordinate for longitude angle obtained 45 
from calibration of said first imaging device; 

compensating said starting point coordinate and said 
ending point coordinate for a radius obtained from 
calibration of said first imaging device. 

U. The method according to claim 9, wherein determin- 50 
ing a second point corresponding to said first point in said 
second marker in said second image further comprises: 

transforming said starting point coordinate from said 
chosen coordinate system to a starting point coordinate 
in a second imaging device coordinate system; 55 

transforming said ending point coordinate from said cho- 
sen coordinate system to an ending point coordinate in 
said second imaging device coordinate system; 

projecting said transformed starting point in said second 60 
imaging device coordinate system on a second image 
sensor plane; 

projecting said transformed ending point in said second 
imaging device coordinate system on said second 
image sensor plane; 65 

determining an intersection point of a third line equation 
passing through said projected starting point and said 



projected ending point projected on said image sensor 
plane and said second marker correlating with said first 
marker in which said first point originated; 
designating said intersection point as said second point 

12. The method according to claim 11, wherein trans- 
forming said starting point coordinate and transforming said 
ending point coordinate in said chosen coordinate system to 
said second imaging device coordinate system comprises: 

applying to said starting point coordinate and said ending 
point coordinate a radius obtained during calibration of 
said second imaging device; 

applying to said starting point coordinate and said ending 
point coordinate a longitude angle obtained from cali- 
bration of said second imaging device; 

applying to said starting point coordinate and said ending 
point coordinate a latitude angle obtained from cali- 
bration of said second imaging device; 

applying to said starting point coordinate and said ending 
point coordinate a roll obtained from calibration of said 
second imaging device; 

applying to said starting point coordinate and said ending 
point coordinate an orientation Cy obtained from cali- 
bration of said second imaging device; 

applying to said starting point coordinate and said ending 
point coordinate an orientation Cx obtained from cali- 
bration of said second imaging device. 

13. The method according to claim 11, wherein projecting 
said starting point and projecting said ending point in said 
second imaging device coordinate system on a second image 
sensor plane further comprises: 

determining a fourth line equation passing through said 
starting point coordinate in said second imaging device 
coordinate system and a second pin hole of said second 
imaging device; 

determining a fifth line equation passing through said 
starting point coordinate in said second imaging device 
coordinate system and said second pin hole of said 
second imaging device; 

determining projection of said transformed starting point 
in said second imaging device coordinate system on 
said second image sensor plane using said fourth line 
equation and a pin bole length of said second imaging 
device; 

determining projection said transformed ending point in 
said second imaging device coordinate system on said 
second image sensor plane using said fifth line equation 
and said pin hole length of said second imaging device. 

14. The method according to claim 1, wherein determin- 
ing said second line equation further comprises: 

determining a position of a second pin hole corresponding 
to said second imaging device capturing said second 
image; 

determining a relative position of said second point with 

respect to said second pin hole; 
determining said second line equation originating from 

said second point and passing through said second pin 

hole. 

15. The method according to claim 1, wherein determin- 
ing intersection of said first line equation and said second 
line equation further comprises: 

determining if said intersection occurs between said first 
line equation and said second line equation; 

if said intersection does not occur then determining a third 
point in said first line equation and a fourth point in said 
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second line equation where a distance between said first 
line equation and said second line equation is at a 
minimum compared with remaining points in said first 
line equation and said second line equation; 

selecting one of said third point and fourth point as an 5 
intersection point. 

16. The method of claim 1, wherein said calibration 
comprises: 

determining a pin hole length of said first imaging device; 10 
determining an orientation C, of said first imaging device; 
determining an orientation C y of said first imaging device; 
determining a roll of said first imaging device; 

determining a latitude of a pin hole corresponding to said ^ 

first imaging device; 
determining a longitude of said pin hole corresponding to 

said first imaging device; and 
determining a radius of said pin hole corresponding to 

said first imaging device. 
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17. The method of claim 16, further comprising: 
determining a pin hole length of said second imaging 

device; 

determining an orientation of said second imaging 
device; 

determining an orientation C y of said second imaging 
device; 

determining a roll of said second imaging device; 
determining a latitude of a pin hole corresponding to said 

second imaging device; 
determining a longitude of said pin hole corresponding to 

said second imaging device; and 
determining a radius of said pin hole corresponding to 

said second imaging device. 

18. The method of claim 17, wherein said pin hole length, 
said orientation said orientation Cj^ said roll, said 
latitude and said longitude are used for compensating said 
starting point and said ending point coordinate. 
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(57) ABSTRACT 

A camera system includes a display monitor which displays 
an image of an object, taken by an optical unit, on a screen 
of the monitor. A reading unit reads a preceding image and 
a current image among a plurality of partially overlapping 
images, from a memory device, the preceding image and the 
current image containing a common element. A determining 
unit determines a positional relation between the preceding 
image and the current image based on a common pattern 
derived from the common element in the two adjacent 
images read by the reading unit. A displaying unit displays 
an image indicating a boundary of the preceding image on 
the screen of the monitor at a shifted position according to 
the positional relation determined by the determining unit, 
with the current image concurrently displayed on the screen 
of the monitor. 

18 Claims, 8 Drawing Sheets 
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SYSTEM AND METHOD FOR DISPLAYING 
AN IMAGE INDICATING A POSITIONAL 
RELATION BETWEEN PARTIALLY 
OVERLAPPING IMAGES 



BACKGROUND OF THE INVENTION 

(1) Field of the Invention 

The present invention relates to a camera system which 
electronically stores an image of an object and displays the 
image on a display monitor. 

(2) Description of the Related Art 

Generally, to achieve an adequately high level of resolu- 
tion of an image captured by using a digital camera or a 
video camera, it is necessary to use a zoom-up function of 
the camera or move the camera close to an object to be 
imaged This makes it difficult to obtain an image covering 
a wide angle related to the object. To capture an image 
covering a wide angle related to the object, it is necessary to 
use a zoom-down function of the camera or move the camera 
away from the object. However, this makes it difficult to 
obtain an image with a high level of resolution. 

In order to obtain a wide-angle image with a high reso- 
lution from an object, a divisional shooting method has been 
proposed. In the divisional shooting method, a plurality of 
partially overlapping images are successively shot so as to 
cover a wide angle related to the object, and they are 
synthesized to create a composite image with an adequate 
level of resolution. 

As disclosed in Japanese Published Utility Model Appli- 
cation No. 8-4783, an image processing device which is 
capable of combining a plurality of partially overlapping 
images together to create a composite image is known. 

To effectively carry out the divisional shooting method, it 
is necessary that, after a preceding image is taken and before 
a current image is taken, the user stop movement of an 
optical axis of the camera at an appropriate position where 
an overlapping portion of the two adjacent images is appro- 
priate for subsequently producing a composite image from 
the images. However, in order to meet this requirement, a 
conventional digital camera requires a special adapter. If 
such an adapter is not used, it is difficult for the conventional 
digital camera to effectively carry out the divisional shooting 
method. In a case of the conventional digital camera with no 
special adapter, there is a possibility that no overlapping 
portion exists between the two adjacent images or a too large 
overlapping portion be produced between the two adjacent 
images. If the overlapping images with undesired overlap- 
ping portions are obtained through the divisional shooting 
method, it is difficult to effectively combine or synthesize 
the images together to create a composite image. 

SUMMARY OF THE INVENTION 

An object of the present invention is to provide a camera 
system which displays an image indicating a positional 
relation among partially overlapping images, and enables an 
operator to easily and effectively carry out a divisional 
shooting process. 

Another object of the present invention is to provide a 
divisional shooting method which displays an image indi- 
cating a positional relation among partially overlapping 
images on a screen of a monitor during a divisional shooting 
mode of a camera system. 

The above-mentioned objects of the present invention are 
achieved by a camera system which comprises: a display 
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monitor which displays an image of an object, taken by an 
optical unit, on a screen of the monitor; a reading unit which 
reads a preceding image and a current image among a 
plurality of partially overlapping images, from a memory 
5 device, the preceding image and the current image contain- 
ing a common element; a determining unit which determines 
a positional relation between the preceding image and the 
current image based on a common pattern derived from the 
common element in the two adjacent images read by the 
10 reading unit; and a displaying unit which displays an image 
indicating a boundary of the preceding image on the screen 
of the monitor at a shifted position according to the posi- 
tional relation determined by the determining unit, with the 
current image concurrently displayed on the screen of the 
monitor. 

15 The above-mentioned objects of the present invention are 
achieved by a divisional shooting method for a camera 
system in which at least two of partially overlapping images 
of an object, taken by an optical unit, are displayed, com- 
prising the steps of: reading a preceding image and a current 
20 image among the partially overlapping images, from a 
memory device, the preceding image and the current image 
containing a common clement; determining a positional 
relation between the preceding image and the current image 
based on a common pattern derived from the common 
25 element in the two adjacent images; and displaying an 
image, indicating a boundary of the preceding image, on a 
screen of a display monitor at a shifted position according to 
the positional relation determined by the determining step, 
with the current image concurrently displayed on the screen 
30 of the monitor. 

In the camera system of the present invention, a positional 
relation between the preceding image and the current image 
is determined based on a common pattern derived from the 
common element in the two adjacent images. The operator 
35 can easily carry out a divisional shooting mode of the 
camera system by viewing both the current image and the 
image indicating the positional relation between the partially 
overlapping images on the screen of the monitor. The 
positional relation between the preceding image and the 
40 current image is clearly noticeable to the operator by view- 
ing the positional relation image on the screen of the monitor 
together with the current image while the camera is panned 
in a desired direction. Therefore, the operator easily stops 
the movement of the optical axis of the camera at an 
45 appropriate position by viewing the positional relation 
image on the screen of the monitor, and turns ON a shutter 
switch to store the current image. 

BRIEF DESCRIPTION OF THE DRAWINGS 
5Q Other objects, features and advantages of the present 
invention will become more apparent from the following 
detailed description when read in conjunction with the 
accompanying drawings in which: 
FIG. 1 is a block diagram of a preferred embodiment of 
55 a camera system of the present invention; 

FIG. 2 is a flowchart for explaining a first example of a 
divisional shooting process performed by a processor of the 
camera system; 
FIG. 3 A and FIG. 3B are diagrams showing an image 
60 which is displayed on a screen of a display monitor when the 
camera is moved in a given direction; 

FIG. 4 is a flowchart for explaining a second example of 
the divisional shooting process performed by the processor 
of the camera system; 
65 FIG. 5 is a flowchart for explaining a third example of the 
divisional shooting process performed by the processor of 
the camera system; 



06/09/2004, EAST Version: 1.4.1 



US 6,466,701 Bl 

3 4 

FIG. 6 is a flowchart for explaining a fourth example of buffer 25 is displayed on the screen 27A of the monitor 27 

the divisional shooting process performed by the processor by the video control unit 24. 

of the camera system; The video control unit 24 further includes a frame buffer 

FIG. 7 is a flowchart for explaining an image storage 26 in addition to the frame buffer 25. The frame buffer 26 

process performed by the processor of the camera system 5 stores auxiliary data indicative of a peripheral boundary 27B 

when a shutter switch is turned ON; and (which will be described later) corresponding to the image 

FIG. 8 A and FIG. 8B are diagrams for explaining a defined in the frame buffer 25. The video control unit 24 

determination of a positional relation between partially displays the peripheral boundary 27B, indicated by the 

overlapping images in the divisional shooting process auxiliary data defined in the frame buffer 26, on the screen 

according to the present invention. 10 27A of the monitor 27, accessing the frame buffer 26 at the 

„ same time as the frame buffer 25. Hence, the image defined 

DETAILED DESCRIPTION OF THE k ^ frame buffer 25 and the auxiliary data defied in the 

PREFERRED EMBODIMENTS frame buffer 26 arc syn(hcsized ^ that the image with the 

A description will now be given of the preferred embodi- peripheral boundary 27B is displayed on the screen 27A of 

ments of the present invention with reference to the accom- 15 the monitor 27 in an overlaid manner. The auxiliary data 

panying drawings. defined in the frame buffer 26 includes a frame number to 

In order to carry out a divisional shooting process, the identify a captured image among a plurality of partially 

present invention utilizes a method and a system for deter- overlapping images, which will be described later. Further, 

mining a positional relation between partially overlapping ^ c auxiliary data may further include image data of a 

images based upon a common pattern in an overlapping 20 displacement vector or a direction of the optical axis of the 

portion of the images. The method and the system are camera, which will be described later, 

disclosed, for example, in U. S. patent application Ser. No. In the camera system of the present embodiment, an 

08/807,571 filed on Feb. 27, 1997 and U. S. patent appli- image memory 28 is a storage device which stores an image 

cation Ser. No. 08/966,889 filed on Nov. 10, 1997, both captured by the video control unit 24. The image memory 28 

assigned to the applicant of the present application. The 25 may be any image storage device, for example, one of 

contents of these co-pending applications are hereby in cor- semiconductor memories including flash memories, or one 

porated by reference. of magnetic disks including floppy disks and mini -disks 

FIG. 1 shows a preferred embodiment of a camera system (MD). 

of the present invention. One example of the camera system 3Q In the camera system of the present embodiment, a 

of the present invention is a digital camera. processor 30 controls the overall operation of the camera 

As shown in FIG. 1, the camera system of the present system and carries out a divisional shooting process includ- 

embodiment includes an optical unit 10. The optical unit 10 m S determination of a positional relation between partially 

has an image pickup device 12, a lens (not shown), and a overlapping images based upon a common pattern in an 

lens positioner (not shown). The image pickup device 12 is 35 overlapping portion of the images. The processor 30 

comprised of a charge-coupled device (CCD). The image includes an arithmetic control unit 32, a read-only memory 

pickup device 12 converts light incident from an object into (ROM) 33, and a random access memory (RAM) 36. The 

an electrical signal, or an image signal indicative of an input ROM 33 stores a number of programs 34A through 34N, and 

image of the object or the scene. The lens positioner ^ xe ^ information, such as character fonts. The arithmetic 

mechanically positions the lens of the optical unit 10 at a 40 contr °l umt 32 carries out individual control operations for 

desired distance from the object along an optical axis of the me elements of the camera system when one of the programs 

lens. Hereinafter, the lens of the optical unit 10 will be 34A through 34N in the ROM 33 is executed by the 

referred to as the camera. processor 30. The RAM 36 is a main memory of the 

In the camera system of the present embodiment, a lens processor 30 which is available to any of the programs when 

positioner actuator 14 actuates the lens positioner of the 45 il is exe cuted- The RAM 36 serves as a work memory 

optical unit 10 so that the lens is positioned at a desired available to the arithmetic control unit 32. Further, the 

distance from the object along the optical axis of the lens. An processor 30 includes a power supply circuit (not shown) 

operation part 16 is an operation part of the camera system whicn su PP lies power to the camera system, and an interface 

of FIG. 1, which includes a mode selection switch 18, a ( not SD0Wn ) which connects the camera system with an 

shutter switch 20, and other control switches (not shown). 50 exterDal host computer. 

An operator can manipulate one of such switches of the In the camera system of FIG. 1, the operator can select 

operation part 16 so as to select one of operational modes of one of the operational modes by using the mode selection 

the camera system or to release the shutter of the camera switch 16. In the present embodiment, the operational modes 

system. of the camera system include a normal shooting mode and 

In the camera system of the present embodiment, a video 55 a ^visional shooting mode, 

control unit 24 converts the signal from the image pickup When the normal shooting mode is selected by the mode 

device 12 into a digital signal, processes the digital signal to selection switch 16, a single image of an object or a scene 

produce a frame of the input image, and stores the frame in is captured through the image pickup device 12, the image 

a frame buffer 25. The frame or image defined in the frame displayed on the screen 27A of the monitor 27 is viewed, and 

buffer 25 is a pixel map that has an array of pixel data, each 60 & c shutter switch 20 is turned ON by the operator so that the 

indicating an intensity (and/or a color value) for a position image defined in the frame memory 25 is stored in the image 

of a corresponding one of the picture elements, or pixels, in memory 28. 

the image. The video control unit 24 displays the image When the divisional shooting mode is selected in the 

defined in the frame buffer 25 on a liquid-crystal display camera system of the present embodiment, a plurality of 

(LCD) monitor 27, accessing the frame buffer 25 as fre- 65 partially overlapping images are successively shot so as to 

quently as a scan rate of the monitor 27. The monitor 27 has cover a wide angle related to an object to be imaged, and 

a display screen 27 A, and the image defined in the frame they are synthesized to create a composite image with an 
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adequate level of resolution. The divisional snooting mode image data to the frame buffer 25 by the video control unit 

is useful to obtain a panoramic image or a bigb-resolution 24. Hence, during the image storage process, the image 

image through image composition. The camera system of displayed on the screen 27A of the monitor 27 is fixed, 

the present invention is particularly relevant to the divisional ^ plocessor 30 at stcp 5503 combines the auxiliary data 

shooting mode, and the followmg description will be given 5 f ^ frame ^ 2fi ^ ft ^ f ^ frame buffef ^ 

of an operation of the camera system of the present embodi- synthesized image, and stores the synthesized 

ment when the divisional shooting mode is performed. . ... . Ao a j j i_ 

In the camera system of the present embodiment, when 10 f t ^ge "ncmory 28 As described above, the 

the divisional shooting mode is selected by the mode selec- dato of the frame buffer 26 includes a frame 

lion switch 20, the processor 30 starts the execution of a in Dun * er !° » captured image among me partially 

divisional shooting processing program 341 among the pro- 10 overlapping images. The auxiliary data of the frame buffer 

grams 34A through 34N in the ROM 33. A divisional * 6 may include other parameter values (which will be 

shooting process is performed by the processor 30 according described later). However, when the image storage process 

to the divisional shooting processing program 341. mth rcs P cct to a first one of pa™ly overlapping images is 

In order to take a first one of partially overlapping images . , P erformed > * c data f the frame buffer 26 is null 

when the divisional shooting process is started, the operator 15 aod only the miage of the frame buffer 25 is stored 

directs the optical axis of the camera (or the lens of the m mc ^ memor y 28 at lhc stc P S502 - 

optical unit 10) to an object to be imaged. In accordance with ^ processor 30 at step S504 allows the writing of image 

the signal from the image pickup device 12, the video data to the frame buffer 25 by the video control unit 24. After 

control unit 24 stores a corresponding frame in the frame 2Q toe ste P S504 is performed, the image storage process of 

memory 25, and displays the image on the screen 27A of the FIG - 7 ends * Hence » after the "»8« storage process is 

monitor 27. The operator turns ON the shutter switch 20 of performed, the image defined in the frame buffer 25 is 

the operation part 16 while viewing the image on the screen displayed on the screen 27A of the monitor 27. 

27A of the monitor 27. A shutter signal from the operation After the first one of the partially overlapping images is 

part 16 is sent to the processor 30 immediately after the 1S taken, the operator pans the camera in a desired direction in 

shutter switch 20 is turned ON. In response to the shutter order to take a following one of the partially overlapping 

signal, the processor 30 stores the image, defined in the images during the divisional shooting mode. By viewing the 

frame memory 25 of the video control unit 24, in the image preceding image with the peripheral boundary on the screen 

memory 28. 27A of the monitor 27, the operator stops the movement of 

The above-mentioned image storage process is performed 30 the °P uca l axis OI * the camera at an appropriate position 

by the processor 30 of the camera system in accordance with where an overlapping portion of the two adjacent images is 

an image storage processing program 34N among the pro- appropriate for subsequently producing a composite image 

grams 34A through 34N stored in the ROM 33. The execu- & om images. Then, the current image is captured and 

tion of the image storage processing program 34N is started stored in the image memory 28 in a similar manner. The 

by the processor 30 in response to the shutter signal. During 35 above-described procedure is repeated until all the partially 

the image storage process, all the image data corresponding overlapping images for the object to be imaged are captured 

to the entire screen 27A of the monitor 27 is not stored in the and stored. In this manner, the partially overlapping images 

image memory 28, but only a portion of the image data are successively shot so as to cover a wide angle related to 

corresponding to an internal portion of the screen 27A of the me object, and they are synthesized to create a composite 

monitor 27 within the peripheral boundary 27B is stored in 40 ima S e ^ an ^equate level of resolution by using the 

the image memory 28. The processor 30 adds a frame technology as disclosed in the above-mentioned U. S. patent 

number to the auxiliary data of the frame buffer 26 and applications. 

stores such data defined in the frame buffer 26, in the image According to the camera system of the present invention, 

memory 28, together with the image defined in the frame the operator can easily carry out the divisional shooting 

buffer 25, during the image storage process. The data being 45 process by viewing both the current image and the periph- 

stored in the image memory 28 may be compressed in a eral boundary 27B (or the preceding image) on the screen 

compact form or may not be compressed in the original 27A of the monitor 27. A positional relation between the 

form. During the image storage process, the writing of preceding image and the current image is clearly noticeable 

image data to the frame buffer 25 is inhibited and the image to the operator by viewing the peripheral boundary 27B on 

displayed on the screen 27A of the monitor 27 is fixed. 50 the screen 27A of the monitor 27 and the current image 

Before the image storage process eods, the writing of image while the camera is panned in the desired direction, 

data to the frame buffer 25 is allowed. Hence, after the image Therefore, the operator easily stops the movement of the 

storage process is performed, the image defined in the frame optical axis of the camera at an appropriate position by 

buffer 25 can be variably updated according to the move- viewing an image of the peripheral boundary 27B, and turns 

ment of the optical axis of the camera, and the resulting 55 ON the shutter switch 20 to store the current image, 

image is displayed on the screen 27A of the monitor 27. FIG. 2 shows a first example of the divisional shooting 

FIG. 7 shows an image storage process performed by the process performed by the processor 30 in accordance with 

processor 30 of the camera system of the present embodi- the divisional shooting processing program 341. 

ment. The image storage processing program 34N among As shown in FIG. 2, at the start of the divisional shooting 

the programs 34A through 34N in the ROM 33 is loaded to 60 process, the processor 30 at step S100 detects whether the 

the RAM 36 and executed by the processor 30 immediately image storage process, shown in FIG. 7, with respect to a 

after the shutter switch 20 is turned ON by the operator. preceding one of the partially overlapping images ends The 

Then, the image storage process of FIG. 7 is performed by en d of the image storage process is notified to the arithmetic 

the processor 30 according to the image storage processing control unit 32 when the execution of the image storage 

program 34N. 65 pr0C essing program 34N has normally ended. When the 

As shown in FIG. 7, at tbe start of the image storage result at the step S100 is negative, the processor 30 repeats 

process, the processor 30 at step S500 inhibits the writing of the step S100. 
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When the result ai the step S100 is affirmative, the direction and a distance for a second number of pixels in the 
processor 30 at step S104 reads out the pixel map of the y-axis direction. Through the above pixel-based method, the 
preceding image from the image memory 28, and reads out processor 30 detects the maximum-similarity common pat- 
the pixel map of a currently^captured image from the frame tern in the two images. That is, the processor 30 at the step 
buffer 25. These pixel maps are temporarily stored in the 5 S106 carries out the determination of the positional relation 
RAM 36. The pixel map of the preceding image is selected between the partially overlapping images, 
as a standard image. Each of the pixel data of the two In mc above -described procedure, the maximum- 
adjacent images corresponding to an overlapping portion of similarity common pattern in the two images is detected by 
me images is divided into blocks of a predetermined size, for thc p i xc l-based method, in order to carry out the 
example, 16 by 16 pixels. 10 determination of the positional relation between the partially 

After the stepS 104 is performed, the processor 30 at step overlapping images. However, according to the present 

S106 performs a matching between corresponding blocks invention, it is also possible to achieve the determination of 

from an overlapping portion of the two adjacent images. a positional relation between partially overlapping images at 

During thc step S106, a common pattern in the two adjacent an accuracy higher than the accuracy of one pixel. As 

images is identified if a certain similarity threshold is met. 15 previously described, the determination of a positional rela- 

This matching may be performed by checking the intensities tion between partially overlapping images based upon a 

of individual pixels of the corresponding blocks. This is common pattern in an overlapping portion of the images are 

useful for reducing the amount of required calculations. disclosed in the above-mentioned U. S. patent applications, 

Alternatively, the matching may be performed by checking and, for that purpose, the divisional shooting process accord- 

the color values of individual pixels of the corresponding 20 ing to the present invention may utilize the method and the 

blocks, but this will increase the amount of required calcu- system. 

lations. The above matching procedures are repeated until all Referring back to FIG. 2, during the step S106, the 

the blocks are processed so that a maximum-similarity processor 30 further determines both coordinates (I, J) of a 

common pattern in the preceding image and the maximum- cent ral pixel of the maximum-similarity common pattern in 

Similarity common pattern in the current image are detected. 25 the preceding ^ coordinates (Im, Jm) of a central 

A method and a system for determining a positional pixel of the maximum-similarity common pattern in the 

relation between partially overlapping images based upon a current image. The coordinates (I, J) and the coordinates 

common pattern in an overlapping portion of the images are (Im, Jm) based on a screen coordinate system of the screen 

disclosed in the above-mentioned U.S. patent applications, 27A of the monitor 27 are determined by the processor 30. 

and the divisional shooting process according to the present 30 The processor 30 at step S108 determines a displacement 

invention utilizes the method and the system. vector (I-Im, J-Jm), which indicates a positional relation 

As previously described, during the step S106 of the between the preceding image and the current image, by the 

divisional shooting process of FIG. 2, a determination of a difference between the coordinates (I, J) and the coordinates 

positional relation between partially overlapping images is 35 (Im, Jm). In the step S108, after the contents of the frame 

carried out. By referring to FIG. 8A and FIG. 8B, a detailed buffer 26 are cleared, the processor 30 writes image data, 

procedure of the determination of the positional relation in indicative of the displacement vector, to the frame buffer 26 

the step S106 will cow be described. as part of the auxiliary data. Hence, the image of the 

It is supposed that the pixel map of the preceding image displacement vector (or the auxiliary data defined in the 

from the image memory 28 and the pixel map of the current ^ frame buffer 26) is displayed on the screen 27A of the 

image from the frame buffer 25 have been read out as in the monitor 27. 

step S104. These pixel maps are temporarily stored in the The processor 30 at step S110 detects whether the opera- 
RAM 36. The pixel map of the preceding image is selected tor stops the movement of the optical axis of the camera (or 
as the standard image. Each of the two adjacent images detects whether the operator turns ON the shutter switch 20). 
corresponding to an overlapping portion of the images is 45 When the result at the step S110 is negative, the above steps 
divided into blocks of a predetermined size. S106 and S108 are repeated. 

As shown in FIG. 8A, pixels "A", "B" and "C" in the When the step S106 is performed for second or subse- 

preceding image and pixels "A m , "B ,w and "C" in the current quent ones of the partially overlapping images, the coordi- 

image correspond to the overlapping portion of the images. nates (I, J) of the central pixel of the maximum-similarity 

During the step S106, a matching between corresponding 50 common pattern in the preceding image and the direction of 

blocks from the overlapping portion of the two adjacent the displacement vector are known. The matching proce- 

images is performed. A common pattern (such as the pixels dures in the step S106 may be performed for only the blocks 

A, B and C and the pixels A', B' and C) in the two adjacent of the current image in the overlapping portion of the two 

images is identified if a certain similarity threshold is met. images, indicated by the direction of the displacement vector 

This matching may be performed by checking the intensities 55 and the coordinates (I, J). By using such a simplified 

of individual pixels of the corresponding blocks. The above matching, the common pattern in the two adjacent images 

matching procedures are repeated until all the blocks are may be identified, and coordinates (Im, Jm) of the central 

processed, so that a maximum-similarity common pattern in pixel of the maximum -similarity common pattern in the 

the preceding image and the maximum-similarity common current image may be determined. 

pattern in the current image are detected. 60 ^ operator stops the panning of the camera at an 
As shown in FIG. 8B, the maximum-similarity common appropriate position where an appropriate overlapping por- 
pattern in the two images is detected if the difference tion of the two adjacent images can be seen with the image 
between the pixel values (or the intensities of the pixels A of the displacement vector on the screen 27A of the monitor 
and A 1 , the pixels B and B' or the pixels C and C) of the 27, and turns ON the shutter switch 20 to store the current 
corresponding blocks is found to be the minimum when the 65 image. Every time the steps S106 and S108 are performed, 
current image is moved relative to the preceding image by the processor 30 compares the currently obtained displace- 
both a distance for a first number of pixels in the x-axis ment vector and the previously obtained displacement vector 
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(stored in ao internal register of the processor 30 or the RAM of the preceding image are not displayed, the operator can 

36) so as to determine whether the operator stops the easily notice a positional relation between the preceding 

movement of the optical axis of the camera. If the difference image and the current image on the screen 27 A of the 

between the two displacement vectors is larger than a monitor 27 with the image 50. 

threshold value, the result at the step S110 is negative. If the 5 FIG. 3B shows an image which is displayed on the screen 

difference between the two displacement vectors is less than 27A of the monitor 27 when the movement of the optical 

the threshold value, the result at the step S110 is affirmative. axis of the camera is stopped and the shutter switch 20 is 

When the result at the step S110 is affirmative, the mrn ^ °*J b * ^ °P?L a ' or - !°. ™' j ™>™ T**?* ° f ^ 

penpnerai counaary *m or. tne preceoing image, to tne - ™ erec tandeABCD< corresponds to an overlapping portion 

frame buffer 26 at a position shifted from the previous of thc two ad; • { descri5ed abo ^ ^ 

position. The shifted position is determined from the P rcv *~ data, indicative of the peripheral boundary 27B of the 

ous position based on the magnitude and direction of the preceding image, is written to the frame buffer 26 at posi- 

displacement vector obtamed in the step S108. Hence, the tions shifted from the previous positions according to the 

image of the peripheral boundary 27B defined in the frame ™ movement of the optical axis of the camera. The image 50 

buffer 26 is displayed on the screen 27A of the monitor 27 0 f me displacement vector corresponding to the magnitude 

as if the peripheral boundary 27B is shifted according to the and direction of the displacement vector is displayed on the 

movement of the optical axis of the camera. screen 27A of the monitor 27. The operator can clearly 

In the step S112, the image data of the displacement notice an appropriate overlapping portion of the two images 

vector obtained in the step S108 may be left in the frame 20 by the image 50 of the displacement vector and the image 52 

buffer 26 without change. Alternatively, the image data of of the peripheral boundary 27B. The image 50 of the 

the displacement vector in the frame buffer 26 may be displacement vector, at the time the movement of the optical 

deleted, and then the image data of the shifted peripheral axis of the camera is stopped, may be displayed on the 

boundary 27B may be defined in the frame buffer 26. The screen 27A of the monitor 27. Alternatively, the display of 

image of the peripheral boundary 27B being displayed on 25 the image 50 of the displacement vector may be omitted, 

the screen 27A of the monitor 27 may be a frame of the FIG.4 shows a second example of the divisional shooting 

preceding image or a solid model of the preceding image process performed by the processor 30 in accordance with 

with a certain color attached to the internal pixels. the divisional shooting processing program 341. 

The operator can easily carry out the divisional shooting 3Q As shown in FIG. 4, at the start of the divisional shooting 

process with the camera system by viewing both the current process in the present embodiment, the processor 30 at step 

image and the peripheral boundary 27B (or the preceding S200 detects whether the image storage process with respect 

image) on the screen 27A of the monitor 27. A positional to a preceding one of the partially overlapping images ends, 

relation between the preceding image and the current image The end of the image storage process is notified to the 

is clearly noticeable to the operator by viewing the periph- 35 arithmetic control unit 32 when the execution of the image 

eral boundary 27B on the screen 27A of the monitor 27 and storage processing program 34N has normally ended. When 

the current image while the camera is panned in a desired the result at the step S200 is negative, the processor 30 

direction. Therefore, the operator easily stops the movement repeats the step S200. 

of the optical axis of the camera at an appropriate position When the result at the step S200 is affirmative, the 

by viewing an image of the peripheral boundary 27A, and ^ processor 30 at step S204 reads out the pixel map of the 

turns ON the shutter switch 20 to store the current image. preceding image from the image memory 28, and reads out 

After the step S112 is performed, the control is transferred the pixel map of the currently -captured image from the 

to the step S100. The processor 30 at the step S100 waits for frame buffer 25. The pixel maps are temporarily stored in the 

the end of the image storage process at which the currently RAM 36. The pixel map of the preceding image is selected 

captured image is further stored in the image memory 28. As 45 as a standard image. Each of the pixel data of the two 

described above, during the image storage process, the adjacent images corresponding to the overlapping portion of 

frame number for the current image and the displacement the images is divided into blocks of a predetermined size, for 

vector for the current image are added to the auxiliary data example, 16 by 16 pixels. 

of the frame buffer 26 and such data defined in the frame After the step S204 is performed, the processor 30 at step 

buffer 26 is stored in the image memory 28 together with the 50 S206 performs a matching between corresponding blocks 

image defined in the frame buffer 25. The frame number and from the two adjacent images. During the step S206, a 

the displacement data are used when synthesizing the par- common pattern in the two adjacent images is identified if a 

tially overlapping images to create a composite image. certain similarity threshold is met. The matching procedures 

FIG. 3A shows an image which is displayed on the screen arc repeated for every block until all the blocks are pro- 

27A of the monitor 27 when the camera is being moved in 55 cessed so that the common pattern in the preceding image 

a given direction indicated in FIG. 3A. In FIG. 3 A, a an d the common pattern in the current image are identified, 

peripheral boundary of a preceding image is indicated by the Further, during the step S206, the processor 30 determines 

dotted-line rectangle A'B'C'D', and a peripheral boundary of both coordinates (I, J) of a central pixel of a maximum- 

a current image is indicated by the solid-line rectangle similarity common pattern in the preceding image and 

ABCD. A displacement between the preceding image and 60 coordinates (Im, Jm) of a central pixel of the maximum- 

the current image proportional to the movement of the similarity common pattern in the current image. The coor- 

optical axis of the camera is defined by the displacement dinates (I, J) and the coordinates (Im, Jm) based on a screen 

vector. In the case of FIG. 3A, the displacement vector is coordinate system of the screen 27 A of the monitor 27 are 

directed to the left and has a length proportional to the determined by the processor 30. 

movement of the optical axis of the camera. An image 50 of 65 The steps S200-S206 in the present embodiment are 

the displacement vector is displayed on the screen 27A of the essentially the same as the steps S10O-S106 in the embodi- 

monitor 27 as indicated in FIG. 3A. Although the contents ment of FIG. 2. 
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The processor at step S208 determines a displacement 30. The sensor 40 may be a built-in type or an external- 
vector (Mm, J-Jm), which indicates a positional relation installation type for the camera system. Other elements of 
between the preceding image and the current image, by the the camera system in the present embodiment are the same 
difference between the coordinates (I, J) and the coordinates as corresponding elements of the camera system shown in 
(Im, Jm). In the present embodiment, during the step S208, 5 FIG. 1, and a description thereof will be omitted, 
the processor 30 writes image data, indicative of the periph- when the divisional shooting mode is selected by the 
eral boundary 27B of the preceding image, to the frame mode selection switch 20, the processor 30 starts the execu- 
buffer 26 at positions shifted from the previous positions. ti on 0 f the divisional shooting processing program 341 in the 
The shifted positions are indicated by the magnitude and ROM 33. The present embodiment of the divisional shooting 
direction of the displacement vector. Hence, the image of the 10 process is performed by the processor 30 according to the 
peripheral boundary 27B denned in the frame buffer 26 is divisional shooting processing program 341. 
displayed on the screen 27A of the monitor 27. ,„ , t tl fi . P „ . 

i.,., • , , . „ , . , In order to take a first one ofpartially overlapping images 

Unlike the embodiment of FIG. 2, during the step S208 in at ^ start of toe divisional shooting process is started, the 

the present embodiment, the processor 30 does not write the operator directs the optical axis of the camera to an object to 

image data of the displacement vector to the frame buffer 26 15 5e and turns 0N ^ switCQ 2 0. A shutter 

as part of the auxiliary data. Hence, the image of the signal from the operation part 16 is sent to the processor 30 

displacement vector is not displayed on the screen 27A of immediately after the shutter switch 20 is turned ON. In 

the monitor 27. response to the shutter signal, the processor 30 reads a signal 

The processor 30 at step S210 detects whether the opera- output by the sensor 40 at that time, and temporarily stores 

tor stops the movement of the optical axis of the camera (or 20 the signal in an internal register of the processor 30 or the 

detects whether the operator turns ON the shutter switch 20). RAM 36. In accordance with the signal from the image 

When the result at the step S210 is negative, the above steps pickup device 12, the video control unit 24 stores a corre- 

S206 and S208 are repeated. sponding frame in the frame memory 25, and displays the 

The operator stops the panning of the camera at an image on the screen 27A of the monitor 27. In response to 
appropriate position where an appropriate overlapping por- 25 the shutter signal, the processor 30 stores the image, defined 
tion of the two adjacent images can be seen with the image in the frame memory 25, in the image memory 28. 
of the displacement vector on the screen 27Aof the monitor The above-mentioned image storage process is performed 
27, and turns ON the shutter switch 20 to store the current by the processor 30 according to the image storage process- 
image. Every time the steps S206 and S208 are performed, ing program 34N in the ROM 33. The execution of the 
the processor 30 compares the currently obtained displace- image storage processing program 34N is started by the 
ment vector and the previously obtained displacement vector processor 30 in response to the shutter signal. During the 
(stored in the internal register or the RAM 36) so as to image storage process, the processor 30 ados both the frame 
determine whether the operator stops the panning of the number and the optical axis direction signal to the auxiliary 
camera. If the difference between the two displacement data of the frame buffer 26, and stores such data defined in 
vectors is larger than a threshold value, the result at the step the frame buffer 26, in the image memory 28, together with 
S210 is negative. If the difference between the two displace- the image defined in the frame buffer 25. During the image 
ment vectors is less than the threshold value, the result at the storage process, the writing of image data to the frame buffer 
step S210 is aflSrmative. 25 is inhibited and the image displayed on the screen 27Aof 

When the result at the step S210 is affirmative, the control AQ the monitor 27 is fixed. Before the image storage process 

is transferred to the step S200. The processor 30 at the step ends, the writing of image data to the frame buffer 25 is 

S200 waits for the end of the image storage process at which allowed. Hence, after the image storage process is 

the currently captured image is further stored in the image performed, the image defined in the frame buffer 25 can be 

memory 28. variably updated according to the movement of the optical 

In the present embodiment, the operator can view a 45 ax * s °f me camera, and the resulting image is displayed on 

peripheral boundary image indicating a positional relation ^ screen 27A of the monitor 27. 

between the current image and the preceding image before After the first one of the partially overlapping images is 

the movement of the optical axis of the camera is stopped or taken, the operator pans the camera in a desired direction in 

the shutter switch 20 is turned ON. The operator can easily order to take a following one of the partially overlapping 

carry out the divisional shooting process with the camera 50 images during the divisional shooting mode. By viewing the 

system, but the current image and the peripheral boundary preceding image with the peripheral boundary on the screen 

image are always displayed on the screen 27Aof the monitor 27A of the monitor 27, the operator stops the movement of 

27. It is desirable that the intensity and/or color of the the optical axis of the camera such that the preceding image 

peripheral boundary image may be set at a suitable value so and the currently-captured image overlap each other with an 

as to prevent the peripheral boundary image from hindering 55 appropriate overlapping portion of the images. Then, the 

the check for the current image on the screen 27A of the current image is captured and stored in the image memory 

monitor 27. 28 together with the auxiliary data, including the frame 

FIG. 5 shows a third example of the divisional shooting number and the optical axis direction signal, in a similar 

process performed by the processor 30 in accordance with manner. The above-described procedure is repeated until all 

the divisional shooting processing program 341. go ^ e P ar tially overlapping images for the object to be imaged 

In the present embodiment, the camera system further are captured and stored, 

includes a three-dimensional gyro sensor 40 connected to With reference to FIG. 5, a description will now be given 

the arithmetic control unit 32 of the processor 30 as indi- of the third example of the divisional shooting process 

cated by the dotted line in FIG. 1. The sensor 40 detects a performed by the processor 30. 

three-dimensional direction of the optical axis of the optical 65 As shown in FIG. 5, at the start of the divisional shooting 

unit 10 and outputs a signal indicating the optical axis process in the present embodiment, the processor 30 at step 

direction to the arithmetic control unit 32 of the processor S300 detects whether the image storage process of FIG. 7 
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with respect to a preceding one of the partially overlapping 
images ends. The end of the image storage process is notified 
to the processor 30. When the result at the step S300 is 
negative, the processor 30 repeats the step S300. 

When the result at the step S300 is affirmative, the 5 
processor 30 at step S304 reads an optical axis direction 
signal (related to the current image) output by the sensor 40 
at that time, and reads the optical axis direction signal 
(related to the preceding image) from the internal register or 
the RAM 36. 10 

After the step S304 is performed, the processor 30 at step 
S306 determines a displacement vector, which indicates a 
positional relation of the preceding image to the current 
image on the screen 27Aof the monitor 27, by the difference 
between the optical axis direction signal related to the 15 
preceding image and the optical axis direction signal related 
to the current image. 

The processor 30 at step S308 writes image data, indica- 
tive of the displacement vector, to the frame buffer 26 as part 
of the auxiliary data after the contents of the frame buffer 26 20 
are cleared. Hence, an image of the displacement vector (or 
the auxiliary data defined in the frame buffer 26) is displayed 
on the screen 27 A of the monitor 27, similar to the image 50 
shown in FIG. 3A and FIG. 3B. 

25 

The processor 30 at step S310 detects whether the opera- 
tor stops the movement of the optical axis of the camera (or 
detects whether the operator turns ON the shutter switch 20). 
When the result at the step S3 10 is negative, the above steps 
S304 through S308 are repeated. 3Q 

The operator stops the panning of the camera at an 
appropriate position where an appropriate overlapping por- 
tion of the two adjacent images can be seen with the image 
of the displacement vector on the screen 27Aof the monitor 
27, and turns ON the shutter switch 20 to store the current 35 
image. Every time the steps S304 through S308 are 
performed, the processor 30 compares the currently obtained 
displacement vector and the previously obtained displace- 
ment vector (stored in the internal register or the RAM 36) 
so as to determine whether the operator stops the movement ^ 
of the optical axis of the camera. If the difference between 
the two displacement vectors is larger than a threshold value, 
the result at the step S310 is negative. If the difference 
between the two displacement vectors is less than the 
threshold value, the result at the step S310 is affirmative. 45 

When the result at the step S310 is affirmative, the 
processor 30 at step S312 writes image data, indicative of 
the peripheral boundary 27B of the preceding image, to the 
frame buffer 26 at positions shifted from the previous 
positions. The shifted positions are indicated by the magni- 50 
tude and direction of the displacement vector obtained in the 
step S306. Hence, the image of the peripheral boundary 27B 
defined in the frame buffer 26 is displayed on the screen 27A 
of the monitor 27. 

In the step S3 12, the image data of the displacement 55 
vector obtained in the step S306 may be left in the frame 
buffer 26 without change. Alternatively, the image data of 
the displacement vector in the frame buffer 26 may be 
deleted, and then the image data of the shifted peripheral 
boundary 27B may be defined in the frame buffer 26. The 60 
image of the peripheral boundary 27B being displayed on 
the screen 27A of the monitor 27 may be a frame of the 
preceding image or a solid model of the preceding image 
with a certain color attached to the internal pixels. 

The operator can easily carry out the divisional shooting 65 
process with the camera system by viewing both the current 
image and the peripheral boundary 27B (or the preceding 
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image) on the screen 27 A of the monitor 27. A positional 
relation between the preceding image and the current image 
is clearly noticeable to the operator by viewing the periph- 
eral boundary 27B on the screen 27A of the monitor 27 and 
the current image while the camera is panned in a desired 
direction. Therefore, the operator easily stops the movement 
of the optical axis of the camera at an appropriate position 
by viewing an image of the peripheral boundary 27B, and 
turns ON the shutter switch 20 to store the current image. 

After the step S3 12 is performed, the control is transferred 
to the step S300. The processor 30 at the step S300 waits for 
the end of the image storage process at which the currently 
captured image is further stored in the image memory 28. As 
described above, during the image storage process, the 
frame number for the current image and the displacement 
vector for the current image are added to the auxiliary data 
of the frame buffer 26 and such data defined in the frame 
buffer 26 is stored in the image memory 28 together with the 
image defined in the frame buffer 25. The frame number and 
the displacement data are used when synthesizing the par- 
tially overlapping images to create a composite image. 

FIG. 6 shows a fourth example of the divisional shooting 
process performed by the processor 30 in accordance with a 
divisional shooting processing program 341. 

As shown in FIG. 6, at the start of the divisional shooting 
process in the present embodiment, the processor 30 at step 
S400 detects whether the image storage process of FIG. 7 
with respect to a preceding one of the partially overlapping 
images ends. The end of the image storage process is notified 
to the processor 30. When the result at the step S400 is 
negative, the processor 30 repeats the step S400. 

When the result at the step S400 is affirmative, the 
processor 30 at step S404 reads an optical axis direction 
signal (related to the current image) output by the sensor 40 
at that time, and reads the optical axis direction signal 
(related to the preceding image) from the internal register or 
the RAM 36. 

After the step S404 is performed, the processor 30 at step 
S406 determines a displacement vector, which indicates a 
positional relation of the preceding image to the current 
image on the screen 27Aof the monitor 27, by the difference 
between the optical axis direction signal related to the 
preceding image and the optical axis direction signal related 
to the current image. 

The processor 30 at step S408 writes image data, indica- 
tive of the peripheral boundary 27B of the preceding image, 
to the frame buffer 26 at positions shifted from the previous 
positions. The shifted positions are indicated by the magni- 
tude and direction of the displacement vector obtained in the 
step S406. Hence, an image of the peripheral boundary 27B 
defined in the frame buffer 26 is displayed on the screen 27A 
of the monitor 27, similar to the image 52 shown in FIG. 3B. 

The processor 30 at step S410 detects whether the opera- 
tor stops the movement of the optical axis of the camera (or 
detects whether the operator turns ON the shutter switch 20). 
When the result at the step S410 is negative, the above steps 
S404 through S408 are repeated. 

The operator stops the panning of the camera at an 
appropriate position where an appropriate overlapping por- 
tion of the two adjacent images can be seen with the image 
of the peripheral boundary 27B on the screen 27A of the 
monitor 27, and turns ON the shutter switch 20 to store the 
current image. Every time the steps S404 through S408 are 
performed, the processor 30 compares the currently obtained 
displacement vector and the previously obtained displace- 
ment vector (stored in the internal register or the RAM 36) 
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so as to determine whether the operator stops the movement 
of the optical axis of the camera. If the difference between 
the two displacement vectors is larger than a threshold value, 
the result at the step S410 is negative. If the difference 
between the two displacement vectors is less than the 5 
threshold value, the result at the step S410 is affirmative. 

The operator can easily carry out the divisional shooting 
process with the camera system by viewing both the current 
image and the peripheral boundary 27B (or the preceding 
image) on the screen 27A of the monitor 27. A positional 10 
relation between the preceding image and the current image 
is clearly noticeable to the operator by viewing the periph- 
eral boundary 27B on the screen 27Aof the monitor 27 and 
the current image while the camera is panned in a desired 
direction. Therefore, the operator easily stops the movement is 
of the optical axis of the camera at an appropriate position 
by viewing the image of the peripheral boundary 27B, and 
turns ON the shutter switch 20 to store the current image. 

When the result at the step S410 is affirmative, the control 
is transferred to the step S400. The processor 30 at the step 20 
S400 waits for the end of the image storage process at which 
the currently captured image is further stored in the image 
memory 28. 

The above -described embodiments of the present inven- 
tion are applied to a digital camera. However, the present 25 
invention is not limited to the above -described embodi- 
ments. It is readily understood that the present invention is 
essentially applicable to a still-video camera and other 
camera systems which electronically store an image of an 
object and display the image on a display monitor. Further, 30 
variations and modifications of the above-described embodi- 
ments may be made without departing from the scope of the 
present invention. 

The present invention is based on Japanese priority appli- 
cation No. 9-245522, filed on Sep. 10, 1997, the entire 
contents of which are hereby incorporated by reference. 

What is claimed is: 

1. A camera system comprising: 

a display monitor for displaying an image of an object, ^ 
taken by an optical unit and stored in a frame buffer, on 
a screen of the monitor; 

a reading unit for reading a preceding image and a current 
image among a plurality of partially overlapping 
images, from the frame buffer, the preceding image and 45 
the current image containing a common element; 

a determining unit for determining a positional relation 
between the preceding image and the current image 
based on a common pattern derived from the common 
element in the two adjacent images read by the reading 50 
unit; and 

a displaying unit for displaying, when a shutter switch is 
turned on, a displaying unit image comprising the 
concurrent display of (i) a boundary image indicating a 
boundary of the preceding image on the screen of the 55 
monitor at a shifted position according to the positional 
relation determined by the determining unit, and (ii) the 
current image, wherein one of the plurality of partially 
overlapping images is defined by the overlap of the 
boundary image and the current image. 60 

2. The camera system according to claim 1, wherein the 
determining unit performs a matching between correspond- 
ing blocks taken from an overlapping portion of the two 
adjacent images, so that a maximum -similarity common 
pattern in the two adjacent images is identified. 65 

3. The camera system according to claim 1, wherein the 
determining unit performs a matching between correspond- 
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ing blocks taken from an overlapping portion of the two 
adjacent images by checking intensities of individual pixels 
of the corresponding blocks. 

4. The camera system according to claim 1, wherein the 
determining unit determines both coordinates of a central 
pixel of a maximum-similarity common pattern in the pre- 
ceding image and coordinates of a central pixel of the 
maximum-similarity common pattern in the current image. 

5. The camera system according to claim 1, further 
comprising: 

a sensor for outputting an optical axis direction signal 
indicating a direction of an optical axis of the optical 
unit; and 

a secondary determining unit for determining a positional 
relation between the preceding image and the current 
image based on a difference between the optical axis 
direction signal output by the sensor with respect to the 
current image and the optical axis direction signal 
output by the sensor with respect to the preceding 
image. 

6. The camera system according to claim 1, wherein the 
determining unit determines a displacement vector, indicat- 
ing a positional relation between the preceding image and 
the current image, based on a difference between coordinates 
of a central pixel of a maximum -similarity common pattern 
in the preceding image and coordinates of a central pixel of 
the maximum-similarity common pattern in the current 
image, and wherein the displaying unit displays an image of 
the displacement vector on the screen of the monitor with the 
current image concurrently displayed on the screen of the 
monitor. 

7. The camera system according to claim 1, further 
comprising an image storing unit for storing an image of the 
object, taken by the optical unit, in an image memory, 
wherein the image storing unit stores auxiliary data, con- 
taining information indicating the positional relation from 
the determining unit, in the image memory, in addition to the 
image stored therein. 

8. The camera system according to claim 5, further 
comprising an image storing unit for storing an image of the 
object, taken by the optical unit, in an image memory, 
wherein the image storing unit stores auxiliary data, con- 
taining information indicating the positional relation from 
the secondary determining unit, in the image memory, in 
additional to the image stored therein. 

9. A divisional shooting method for a camera system in 
which at least two of partially overlapping images of an 
object, taken by an optical unit and stored in a frame buffer, 
are displayed, comprising the steps of: 

reading a preceding image and a current image among the 
partially overlapping images, from the frame buffer, the 
preceding image and the current image containing a 
common element; 

determining a positional relation between the preceding 
image and the current image based on a common 
pattern derived from the common element in the two 
adjacent images; and 

displaying, when a shutter switch is turned on, a display- 
ing unit image comprising the concurrent display of (i) 
a boundary image indicating a boundary of the preced- 
ing image on a screen of a display monitor at a shifted 
position according to the positional relation determined 
by the determining step, and (ii) the current image, 
wherein one of the at least two partially overlapping 
images is defined by the overlap of the boundary image 
and the current image. 
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10. The method according to claim 9, wherein, in the 
determining step, a matching between corresponding blocks 
taken from an overlapping portion of the two adjacent 
images is performed, so that a maximum-similarity common 
pattern in the two adjacent images is identified. 

11. The method according to claim 9, wherein, in the 
determining step, a matching between corresponding blocks 
taken from an overlapping portion of the two adjacent 
images is performed by checkiog intensities of individual 
pixels of the corresponding blocks. 

12. The method according to claim 9, wherein, in the 
determining step, both coordinates of a central pixel of a 
maximum -similarity common pattern in the preceding 
image and coordinates of a central pixel of the maximum- 
similarity common pattern in the current image are deter- 
mined. 

13. The method according to claim 9, further comprising 
the steps: 

outputting an optical axis direction signal indicating a 
direction of an optical axis of the optical unit; and 

determining a positional relation between the preceding 
image and the current image based on a difference 
between the optical axis direction signal output by the 
sensor with respect to the current image and the optical 
axis direction signal output by the sensor with respect 
to the preceding image. 

14. The method according to claim 9, wherein, in the 
determining step, a displacement vector, indicating a posi- 
tional relation between the preceding image and the current 
image, is determined based on a difference between coor- 
dinates of a central pixel of a maximum-similarity common 
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pattern in the preceding image and coordinates of a central 
pixel of the maximum-similarity common pattern in the 
current image, and wherein, in the displaying step, an image 
of the displacement vector is displayed on the screen of the 
5 monitor with the current image concurrently displayed on 
the screen of the monitor. 

15. The method according to claim 9, further comprising 
a step of storing an image of the object, taken by the optical 
unit, in an image memory, wherein auxiliary data, containing 
information indicating the positional relation from the deter- 
mining unit, is stored in the image memory in addition to the 
image stored therein. 

16. The method according to claim 13, further comprising 
15 a step of storing an image of the object, taken by the optical 

unit, in an image memory, wherein auxiliary data, containing 
information indicating the positional relation from the sec- 
ondary determining unit, is stored in the image memory, in 
additional to the image stored therein. 

20 17. Hie camera system according to claim 1, wherein the 
displaying unit displays a displacement vector, correspond- 
ing to the positional relation between the preceding image 
and the current image, during movement of an optical axis 
of the optical unit. 

2S 18. The method according to claim 9, further comprising 
the step of displaying a displacement vector, corresponding 
to the positional relation between the preceding image and 
the current image, during movement of an optical axis of the 
optical unit. 

30 

♦ * * * * 
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(57) ABSTRACT 

A system and method are described for aligning a locally- 
reconstructed three-dimensional object, whose local recon- 
struction is relative to a local coordinate system, to a global 
coordinate system by using pre-mapped control points 
which are projected onto one or more of the images that may 
be used to generate the local reconstruction. A system 
includes a control point information generator and an align- 
ment generator. The control point information generator is 
configured to identify in at least image associated with a 
local reconstruction of the object a projection of at least one 
control point in the scene onto the at least one image, and 
generate local projected coordinate information indicating 
coordinates of the projection in the at least one image. The 
alignment information generator is configured to utilize the 
local projected coordinate information and mapping infor- 
mation relating to global coordinate information indicating 
coordinates of the at least one control point relative to the 
global coordinate system to generate alignment information 
relating a local coordinate system associated with the local 
reconstruction to the global coordinate system. 

24 Claims, 4 Drawing Sheets 
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100. SELECT A SET OF CONTROL POINTS 



] 
] 



101. DETERMINE COORDINATES OF CONTROL POINTS 
RELATIVE TO THE GLOBAL COORDINATE SYSTEM 



110. OPTICAL HEAD RECORDS ONE OR MORE IMAGES OF 
THE SCENE 



J 



I 



111. OPTICAL HEAD PROVIDES THE RECORDED IMAGES 
TO THE IMAGE PROCESSOR 



112. IMAGE PROCESSOR PROCESSES THE IMAGES TO 
IDENTIFY THE CONTROL POINTS FROM THE SET OF 
CONTROL POINT, PROJECTIONS OF WHICH HAVE BEEN 
RECORDED IN THE RESPECTIVE IMAGES 



i13. IMAGE PROCESSOR DETERMINES 
THREE-DIMENSIONAL COORDINATES OF CONTROL 
POINTS IN THE LOCAL COORDINATE SYSTEM DEFINED 
BY THE CURRENT POSITION AND ANGULAR 
ORIENTATION OF THE OPTICAL HEAD 



6 



FIG. 2 
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1 14. IMAGE PROCESSOR SELECTS ONE OF THE THREE 
IMAGES 



115. IMAGE PROCESSOR DETERMINES WHETHER THERE 
ARE AT LEAST THREE CONTROL POINTS FROM THE SET 
OF CONTROL POINT PROJECTED ONTO THE SELECTED 
IMAGE 



h-o 
] 



1 16. IMAGE PROCESSOR GENERATES g(f J) FOR 
RESPECTIVE PAIRS OF CONTROL POINTS 



117. IMAGE PROCESSOR GENERATES q(ij) FOR 
RESPECTIVE PAIRS OF CONTROL POINTS 



1 18. IMAGE PROCESSOR SETS UP THE SYSTEM OF 
EQUATIONS (16) 



J 



119. IMAGE PROCESSOR SOLVES THE SYSTEM TO 
DETERMINE THE VALUES OF THE SCALARS 
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FIG.2B 



4 NO 



120. IMAGE PROCESSOR DETERMINES WHETHER ONE 
OR TWO CONTROL POINTS WERE PROJECTED ONTO THE 
SELECTED IMAGE 



YES 

± 



i 



121 . IMAGE PROCESSOR SELECTS ONE OF THE POINTS 
AND DETERMINES WHETHER THE SELECTED CONTROL 
POINT WAS ALSO PROJECTED ONTO ONE OR MORE OF 
THE OTHER IMAGES. 



^122. IMAGE PROCESSOR GENERATES VALUES OF Q AND 
Q 1 AND d-d 1 THEREFOR FOR THE RESPECTIVE IMAGES, 
ALONG WITH Q** AND d-d" IF THE CONTROL POINT WAS 
PROJECTED ONTO THE THIRD IMAGE 



} 



^23. IMAGE PROCESSOR USES EQUATION 16, OR 
EQUATION 17 IF THE CONTROL POINT WAS PROJECTED 
ONTO THE THIRD IMAGE, TO DETERMINE VALUES FOR 
THE RESPECTIVE SCALARS 



124. IMAGE PROCESSOR DETERMINES WHETHER THERE 
IS ANOTHER CONTROL POINT PROJECTED ONTO THE 
SAME IMAGE THAT HAS NOT BEEN PROCESSED 



) 
> 



125. IMAGE PROCESSOR DETERMINES WHETHER IT HAS 
SELECTED ALL OF THE IMAGES RECORDED BY THE 
OPTICAL HEAD IN STEP 110. 



YfiS 



ho 



126. IMAGE PROCESSOR DETERMINES ROTATION 
MATRIX R AND TRANSLATION MATRIX t 
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SYSTEM AND METHOD FOR ALIGNING A 
LOCALLY-RECONSTRUCTED THREE- 
DIMENSIONAL OBJECT TO A GLOBAL 
COORDINATE SYSTEM USING PARTIALLY- 
DETECTED CONTROL POINTS 

This application claims continuing data of No. 60/142, 
347, Jun. 28, 1999. 

FIELD OF THE INVENTION 

This invention relates generally to the field of reconstruct- 
ing and/or manipulating surface geometries of one or more 
three dimensional objects in a scene, from a plurality of two 
dimensional images of the scene, and more particularly to a 
system and method for relating a local coordinate system 
associated with a reconstruction of at least a portion of the 
scene to a global coordinate system. 

BACKGROUND OF THE INVENTION 

Reconstruction and/or manipulation (generally, 
"reconstruction") of surface features of three-dimensional 
object(s) in a scene, from a plurality of two-dimensional 
images of the object(s), is useful in a number of applications. 
U.S. patent application Ser. No. 08/989,047, filed Dec. 11, 
1997, in the names of Dan Albeck et al., and entitled 
"Apparatus And Method For 3-Dimensional Surface Geom- 
etry Reconstruction," now U.S. Pat. No. 5,617,151, issued 
Dec. 26, 2000, (hereinafter referred to as the "Albeck 
application") describes an apparatus for performing such 
reconstruction using a rig including an optical head of three 
cameras, using a tensor arrangement described in U.S. patent 
application Ser. No. 08/497,224, filed Jun. 30, 1995, in the 
name of Amnon Shashua, and entitled "Apparatus And 
Method For Recreating And Manipulating A 3D Object 
Based On A 2D Projection Thereof* now U.S. Pat. No. 
5,821,943, issued Oct. 13, 1998, (hereinafter referred to as 
the "Shashua application") to generate information regard- 
ing reconstruction for the features of the object(s) from three 
images generated by the cameras. In the arrangement 
described in the Shashua application, the surface features 
that are reconstructed are defined by points that have coor- 
dinates in a coordinate system relative to one of the cameras 
in the rig. A problem arises in reconstruction if the surface 
features that are to be reconstructed cannot all be recorded 
by all of the cameras with the rig in one position. The 
apparatus described in the Albeck application provides a 
mechanism for moving the rig so as to allow the cameras to 
record sets of images of various portions of the surfacc(s) of 
the object(s). However, when the rig moves from one 
location, in which the cameras record a set of images of one 
portion of the object(s), to another location, in which the 
cameras record another set of images of another portion of 
the object(s), the coordinate system for the points defining 
the surface features of the various portions of the object(s) 
also changes. 

In order to utilize the reconstruction information gener- 
ated in the two "local" coordinate systems in a unitary 
manner in connection with the object(s) in the scene, it is 
necessary to relate the local coordinate systems to a common 
global coordinate system, which will allow all of the points 
of the various portions of the reconstructed object(s) to be 
related to the global coordinate system, effectively "stitch- 
ing" the reconstructions together. The global coordinate 
system can conveniently be one of the two local coordinate 
systems that were used in the reconstruction, or it can be a 
third coordinate system, but in any case all of the points for 
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the various portions of the reconstructed object need to be 
related to the global coordinate system. When a rig, such as 
the rig described in the Albeck application is moved from 
one position to another, to facilitate recording of sets of 

5 images of different portions of the object(s), the movement 
comprises one or both of a translation and a rotation of the 
rig, both in three dimensions. If the translational and rota- 
tional movement of the rig can be controlled sufficiently 
precise, the relation of the coordinate system after move- 

10 ment to the coordinate system before movement can easily 
be determined. However, if, for example, the mass of the rig 
is sufficiently large, in a number of applications the move- 
ment of the rig cannot readily be controlled sufficiently 
precisely to allow sufficiently precise reconstruction. 

15 U.S. patent application Ser. No. 09/165,687, filed Oct. 2, 
1998, in the name of Tamir Shalom, et al., and entitled 
"System And Method For "Stitching" A Plurality Of Recon- 
structions Of Three-Dimensional Surface Features Of 
Object(s) In A Scene Defined Relative To Respective Coor- 

20 dinate Systems To Relate Them To A Common Coordinate 
System," now U.S. Pat. No. 6,201,541, issued Mar. 13, 
2001, (hereinafter referred to as the Shalom application) 
describes an arrangement for relating a reconstruction of a 
portion of a scene to a global coordinate system. In the 

25 arrangement described in the Shalom application, a recon- 
struction of one portion of a scene is generated, which is 
related to the global coordinate system. Reconstructions of 
other portions can be related to the global coordinate system, 
but in the arrangement described in the Shalom application, 

30 the respective pairs of reconstructed portions will need to 
overlap with each other. That is, reconstructions succes- 
sively displaced from the reconstruction which was origi- 
nally related to the global coordinate systems will need to 
overlap, at least to some extent, with the overlapping por- 

35 tions providing information that is used to relate the local 
coordinate systems of the overlapping portions, thereby 
facilitating relation of the sequence of local coordinate 
systems to each other and, ultimately, to the global coordi- 
nate system. However, it will be appreciated that errors can 

40 develop in relating local coordinate systems of successive 
pairs of reconstructions, which can result in successively 
increasing errors in relating the local coordinate systems of 
the respective reconstructions to the global coordinate sys- 
tem. In addition, when the optical head is moved from one 

45 location to another to facilitate recording images from which 
the respective reconstructions are generated, the movement 
needs to be rigid, that is, the cameras need to maintain their 
respective orientations with respect to each other. If the 
optical head is not rigid, that is, if the cameras change their 

50 translational or rotational orientations with respect to each 
other, errors in relating the local coordinate systems of the 
respective pairs of reconstructions can also develop. 

SUMMARY OF THE INVENTION 

55 The invention provides a new and improved system and 
method for aligning a locally-reconstructed three- 
dimensional object to a global coordinate system using 
partially-detected control points. 

In brief summary, the invention provides a system and 

60 method for aligning a locally-reconstructed three- 
dimensional object, whose local reconstruction is relative to 
a local coordinate system, to a global coordinate system by 
using pre-mapped control points which are projected onto 
one or more of the images that may be used to generate the 

65 local reconstruction. A system in accordance with the inven- 
tion includes a control point information generator and an 
alignment generator. The control point information genera- 
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tor is configured to identify in at least image associated with dimensional images in a manner similar to that described in 

a local reconstruction of said object a projection of at least the aforementioned Shashua application, 

one control point in the scene onto said at least one image, The system 10 is further provided with a motor 16 which, 

and generate local projected coordinate information indicat- under control of a control processor 17, can move the rig U 

ing coordinates of said projection in said at least one image. 5 to facilitate direction of the cameras 12(c) to another portion 

The alignment information generator is configured to utilize of the scene 14, including respective other portion(s) of the 

said local projected coordinate information and mapping surfaces) of respective object(s) in the scene 14 from which 

information relating to global coordinate information indi- position the cameras 12(c) can record a second set of 

eating coordinates of said at least one control point relative images. The rig 11 before one such illustrative movement, 

to said global coordinate system to generate alignment 10 indicated in FIG. 1 by the arrow associated with reference 

information relating a local coordinate system associated numeral 18, is depicted in solid lines in FIG. 1, and rig U 

with said local reconstruction to said global coordinate after the illustrative movement 18 depicted in dashed lines 

system. in FIG. 1. In addition, the set of images recorded by the 

After the alignment information has been generated, the respective cameras 12(c) prior to the illustrative movement 

reconstruction in the local coordinate system can be aligned 15 18 of rig 11 is represented by images 20(AX0) through 

to the global coordinate system. This can be done with a 2( K A X C ) (generally identified by reference numeral 20(A) 

plurality of local reconstructions, essentially stitching them ( c ))' and ^ xi of images recorded by the respective 

together. cameras 12(c) after the illustrative movement 18 of rig 11 is 

represented by images 20(B)(0) through 20(BXQ (generally 

BRIEF DESCRIPTION OF THE DRAWINGS 20 identified by reference numeral 20(BXc)). The respective 

Tins invention is pointed out with particularity in the ^ ^T^ 2 ^ ^°?R™ ^V^u 

appended claims. The above and further advantages of this caaa ^ n < *? "^^^M^?^ 

invention may be better understood by referring to the ^ l ^ ° f T 20(AXc) and 20(B)(c) will 

following description taken in conjunction with the accom- P\ ef ^ ably * e of d ?* reDt ^T* r f?*"™ sm ^ 
panying drawings, in which- 25 (s) ' ^ porUons of ^ res P ectlve surface(s) represented by 

1 T . ,, / • , i. • the sets of images 20(A)(c) and 20(BXc) may, but need not, 

FIG. 1 schematically depicts a system for aligning a overlap 

locally-reconstructed three -dimensional object to a global ^ e ^ ^ „ ^ ^ . ( )( } 

coordmate system using partiaHy-detected control points, ided * f hQ camfiras \ m J m JJ^ 

constructed m accordance with the invention: in L Mmfln . ie , A .u .u j- i c 

1 30 movement 18 to generate the three-dimensional surface 

FIG. 2 is a flow chart depicting operations performed by information relating to the portions) of the object(s) 

the system in connection with aligning a locally- depicted by the images 20(A)(c). In addition, the image 

reconstructed three-dimensional object to a global coordi- processor 15 processes the images 20(BXc) provided by the 

nate system using partially -detected control points. cameras 12(c) after the illustrative movement 18 to generate 

DETAILED DESCRIPTION OF AN 35 ^ e tDree "di mens i° na l surface information relating to the 

ILLUSTRATIVE EMBODIMENT portion(s) of the object(s) depicted by images 20(BXc). In 

one embodiment, operations performed by the image pro- 

FIG. 1 is a diagram of a system 10 for aligning a cessor 15 in generating the' three-dimensional surface 
locally -reconstructed three-dimensional object to a global information, in connection with the images recorded both 
coordinate system using partially-detected control points, 40 before and after the illustrative movement 18, correspond to 
constructed in accordance with the invention. With reference those operations described in the aforementioned Shashua 
to FIG. 1, system 10 includes a rig 11 that supports an optical application, and the information so generated may represent 
head 12 that comprises a plurality of cameras 12(0) through a three-dimensional reconstruction of the portion(s) of the 
!2(C) (generally referred to by reference numeral 12(c)) objects) depicted in the respective set of images. In that 
mounted on a common support 13. The cameras 12(c) are 45 embodiment, the image processor 15 generates, from a 
preferably focused on generally the same portion of a scene, plurality of sets of points p y , p/ and p," ("j" being an index) 
including the same portions) of the surface(s) of one or in images 20(AX0) through 20(A)(2), respectively, with 
more objects generally identified by reference numeral 14 in points p yi pj and p/ in each set corresponding to projections 
the scene, thereby to facilitate recording of a set of two- of a point P y on a surface of an object in the scene 14 onto 
dimensional images of the respective portion(s) by the 50 the respective images 20(A)(0) through 20(AX2), a tensor 
respective cameras 12(c). The cameras 12(c) provide the set T^. In generating the tensor T Ai the image processor 15 uses 
of recorded images to an image processor 15 which two-dimensional coordinates, in the respective images 
generates, from the images provided thereto by the cameras 20(AXc), of a selected number of sets of respective points p., 
12(c), information relating to a three-dimensional recon- p/ and p/. As described in the Shashua application, seven 
stmction of surface features of the respective object(s) 14 in 55 sets of points p /t pj and p/ will suffice to enable the image 
the scene, for the respective portion(s) to which the cameras processor 15 to generate the tensor T A . As also described in 
12(c) are focused. In one embodiment, the rig 11 and optical the Shashua application, using the tensor T^ and two- 
head are similar to the respective elements of the apparatus dimensional coordinates of any set of points p., p.' and p." in 
described in the aforementioned Albeck application. In that the images 20(AXc) which are projections of ' a point P 
embodiment, the cameras 12(c) preferably include CCD 60 (including, but not limited to, the seven sets of points used 
("charge-coupled device") image recording devices, which to generate the tensor T A ), the image processor 15 can 
provide image information in digital form to the image determine the three-dimensional coordinates of the corre- 
processor 15 for processing. In addition, in that sponding point P, in the scene 14. The three dimensional 
embodiment, the image processor 15 includes suitably- coordinates ofpointP y will be in a coordinate system relative 
programmed computing devices (such as a suitably- 65 to one of the images 20(A)(c), illustratively the image 
programmed general purpose computer) that generate the 20(A)(0) recorded by camera 12(0) before the illustrative 
three-dimensional surface information from the set of two- movement 18. 
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Similarly, the image processor generates, from coordi- It will be appreciated that, if the position and angular 

nates a plurality of points q /f q/ and q," in images 20(B)(0) orientation of the rig U before and after the illustrative 

through 20(BX2), respectively, all corresponding to projec- movement 18 can be controlled or determined (by, for 

tions of respective points Q y onto the respective images example, mechanical or electronic sensors) to a sufficient 

20(BX0) through 20(B)(2), a tensor and coordinates for 5 degree of accuracy they (that is, the position and angular 

the respective points Q ; - in a coordinate three-dimensional orientation) can be used to define the translational and 

coordinate system relative to one of the images, illustra- rotational components to relate the coordinate system after 

tively image 20(BX0), recorded by camera 12(0) after the the illustrative movement 18 to the coordinate system before 

illustrative movement 18. If optical head 12 is rigid under the illustrative movement 18. However, in one embodiment, 

the movement, so that the cameras 12(c) will be in the same to provide a higher degree of accuracy, the image processor 

positions relative to each other before and after the illustra- 15 for each reconstruction uses information relating to 

tive movement 18, and if the three-dimensional coordinate respective control points (not separately shown) that are 

systems are relative to images recorded by the same camera distributed throughout the scene, images of at least some of 

12(0), then the tensor T B will correspond to the tensor T^. which may be recorded one or more of the images 20(A)(c) 

On the other band, if the optical head 12 is not rigid under and/or 20(B)( C ) in each set. Generally, separately from the 
the illustrative movement 18, the tensor T g will differ from 15 recording of the sets of images before and after the illus- 

the tensor T A . trative movement 18, during a mapping stage information is 

As noted above, the coordinates generated by the image generated as to the positions of the control points relative to 

processor 15 for points P, generated using images 20(A)(c) the global coordinate system. That information, along with 

are defined relative to a three-dimensional coordinate system information about the ones of the control points recorded on 
associated with one of the cameras 12(c), for example, 20 respecuVe ones of images 20(A)(c) can be used to facilitate 

camera 12(0), in its position^ prior to the illustrative move- relation of the loca] coordinate system ^ the 

Zll ^ °P tical head 12 * the various positions to the global 

dinates generated by the image processor 15 for points Q, t * „„. ~ t . . A . r 

from images 20(B)(c) are defined relative to a coordinate coordinate system. Operations performed by the image 
system that is preferably associated with the same camera 25 pr0CeSSOr 15 " ^ termimng the translational and rotationa 

12(0), but which will be relative to the camera's position com P ODe t Dts relatul g a J** coordinate system to the global 

after the illustrative movement 18 at which image 20(B)(0) S^?™! * . * ^ nbed 10 connection with 

was recorded. In each case, the coordinates will be denned ^wchart depicted in RG. 2. Before proceeding to a descrip- 

relative to two coordinate axes (illustratively referred to T °* mQ ' 2 DV WaV ° f background, given a set of 

herein as «x" and "y" axes") which correspond to the plane 30 control points {G,-},^ in the scene, coor- 

of the respective image 20(A)(0) and 20(B¥0), and a third dm *?f m ** gl ° bal coordma i e system for which are mea- 

axis (illustratively referred to herein as a V' axis) that is SUfed m n *™ m Z ™ e three dimensional control 

transverse thereto. It will be appreciated that the coordinate ^/£'fc.i?^^ W u "^ H fT \H 

systems for the two sets of points P, and Q, in the scene 14 * t ? S ™M e )\ although as will be described 

wiU differ. To be able to utilize both sets of points in a 35 * el ° W ' DOt ^control pomts need have projections in all 

unitary fashion, it is desirable to associate them both to a & ree ^ a g<*- Assume further that the jrojecUon matrices for 

common global coordinate system. The particular global ? C res pcctive images are represented by T T and T\ each 

coordinate system that is selected is nof important. For ^ * ^J 1 ^' A™****™ lhat lhe local 

simplicity and without loss of generality, the global coordi- <;°° rdmale SVJ *em dlffers from * e global coordinate system 

nate system can be selected to £ the local coordinate system 40 by 3 TT * t * * anslat i° n In J hat case ' durin S lhe 

before the illustrative movement 18 or after the iUustrative ^^ 0 ° V ^ coordinates of the rx>mtsp p' and 

movement 18. On the other hand, the global coordinate P .< ° f J° COQtro1 p0lQts t G '>'-* m the images 20(A)(c) are 

system may comprise a coordinate system separate from any glVCD ^ 

of the local coordinate systems, illustratively, coordinate r Q 

system 21. 45 T . * .*M (1) 

Generally, the relation between the global coordinate LOOOiJUJ I 1 J 

system and the local coordinate systems consists of one or r r t ] r C, 1 , [ pM (2) 

both of a translational component and/or a rotational r "|o 0 0 1 j'[ 1 J = A '[ 1 J 

component, both in three dimensional space. Thus, if, the L 

coordinate system associated with camera 12(0), both before 50 ^ 
and after the illustrative movement 18, has an origin at the 

center of the image plane of the camera 12(0), the transla- r ■ f * ' ] . f Ci 1 = Af[ P '] (3) 

tional component comprises the position of the center of the 1 0 0 0 1 J [ l J ' I 1 J 
image plane after the illustrative movement 18, in the 

coordinate system associated with the camera 12(0) before 55 respectively. It will be appreciated that the matrix 

the illustrative movement 18. In addition, the rotational J FP 

component corresponds to the angle of the image plane after r R 

the illustrative movement 18, relative to the image plane 0001 

before the illustrative movement 18. It will be appreciated J 

that, if the position of the origin after the illustrative move- 60 

ment 18 corresponds to the position of the origin before the in equations (1) through (3) effectively operates to transform 
illustrative movement 18, there is no translational compo- the coordinates of the control points {G} . " from the global 
nent for the illustrative movement 18. Similarly, if the coordinate system to the local coordinate"system following 
angular position of the coordinate axes after the illustrative the transformation, and multiplying by the respective pro- 
movement 18 corresponds to the angular position of the 65 jection matrix T, T and T provides the coordinates of the 
coordinate axes before the illustrative movement 18, there is projections in the respective images 20(A)(c), up to a scale 
no rotational component for the illustrative movement 18. factor represented by the respective scalars V- and \ n - 
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If ihe projection matrices T, T and T" are decomposed 
into respective three by three and three by one sub-matrices 
T=[S s], T^S' s'] and T"=[S" s"], where sub-matrices S, S' 
and S" are three-by-three matrices and sub-matrices s, s' and 
s" are one-by-three matrices (or vectors) equations (1) 
through (3) can be rewritten as 



5-(ff»Gi+0 + «aAi[*J 



(4) 



(5) 



10 



respectively. Since the matrices S, S', and S" are invertible, 
equations (4) through (6) can be re-written as 



R • C; + 1 = \,s~ 



and 



(7) 
(8) 

(9) 



R-G^\ i P r d^\'^ r d' r \' ( "P\-d' i 



20 



25 



30 



respectively. Defining sets of three-dimensional points 
{p£*}i-i"i {P.W and {P",.}^ by 



35 



and further defining d-S^s, tf-S'" 1 ^' and d"*^"" 1 ^", and 
noting that the left sides of equations (7) through (9) all 
equal R-G.+t, those equations (7) through (9) can be 
re-written 



(10) 



45 



Since the values of scalars {X,. {V,}^," and {V,} w " 
are unknown, the points P., P, and P" f can be normalized to 

Q, = — , 0' = -3_ and or - 



50 



where ||PJ|, ||PJ| and ||P"J| represent the lengths of the vectors 
representing the points P„ P, and P M „ respectfully in the 
local coordinate system, and in that case equation 10 can be 
re-written as 

R-Gt+t-^Qrd-k'grd'-K-d' (l l) 

Using equation (10), the values of the scalars {Xj^j", 
{^/li-i" and {X",},-.!" and the components of the rotation 
matrix R and the translation matrix t can be obtained by 
minimizing the cost function 



55 



60 



8 

-continued 

iK* • d + 1) - (a;g; - <nn 2 + \\{r . c, + o - (a: or - on 2 . 



Note that, if the rotation matrix R and translation matrix 
t are known, which may be the case if the position and 
orientation of the local coordinate system relative to the 
global coordinate system are known, the values of the 
scalars {X,} (-1 W , {XVL_i" ^ ^"/h-i" can ^ determined by 
projecting the point R G^+t onto respective lines X,Q,-d, 
Vfi' r & and X'VQVd"- 1° that case, and since Q r Q=Q ,7 ^Q'= 
Q" r Q"=l, where "T* represents the transpose operation, 



(6) 15 



and 



(13) 

(14) 
(15). 



(12) 



65 



On the other hand, if the values of the scalars {X,},^", 
{Kiii-i n {X"/},./ 1 are known, the rotation matrix R and 
translation matrix t can be determined using the well-known 
Singular Value Decomposition (S VD) algorithm that fits the 
sets of points {X / Q 1 -d},._ 1 , \ {^ffr<V} M n and {X fl i Q' , l - 
d"h_i n t0 tne set of control points {Gi} iml n 9 repeated three 
times for each of the three sets. 

Accordingly, one can start with the values of the rotation 
matrix R and translation matrix t and use them to determine 
the values of the scalars {X,}^", {V,}**" and {X",-}^", or 
start with the values of the scalars {X t ) i _ 1 n , {k\} iml n and 
and use them to determine the values of the 
rotation matrix R and translation matrix t. In one 
embodiment, the image processor initially determines values 
of the scalars {X,}^", {X',.}^" and {X" f } | . -J n and uses them 
to determine the values of the rotation matrix R and trans- 
lation matrix t. Initially, it determines the values of the 
scalars {h}^ {Kh-i n and using the fact that the 

distance between two points in the scene is invariant when 
the optical head undergoes a rigid transformation (although 
the distance between the projections of the points in respec- 
tive images may change) as follows. If projections of two 
different control points G, and G ; appear in the same image, 
the distance between the points is given by ||G J -G / |H|(R-G ; + 
t)-(RG,+t)||. Since (R G l +t)-(X l Q J -d) and QL-GM)-QJQ r 

On the other hand, if a control point G, in the scene is 
projected onto two images, then the values of the scalars 
(Mi-i" and {X' 4 }^ a " associated with the respective images 
can be determined using the fact that X i Q r -d-X , i Q' r cT for the 
two images. This provides three equations for the two 
unknowns X,- and X'^, which can be solved in a conventional 
manner; 

A third case is that each of two different control points G, 
and Gy in the scene projects onto one of the two images. In 
that case, the distance between the two points is given by 
llGrGyll-IKR-G^tHR-Q^+t)!!, in which case UG.-GJHI 
(X,Q l -d)-(X t / Q > -d , ||. However, since this case is less useful 
than the two described above, it will not be considered 
further. 

Considering the first case, that is, the case in which 
projections of two different control points G,- and G y appear 
in the same image, as noted above, the distance between the 
points is given by ||G,.-G y JJH|XA--X > Q / ||. Defining g^G,- 
9/) r (Gr G /) and q lV -Qf r Q /f that equation can be re-written 



(36). 
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In the case of "nT points, use of equation (16) provides in the name of Amnon Shashua, and entitled "System And 

Method For Reconstructing Surface Elements Of Solid 
Objects In A Three -Dimensional Scene From A Plurality Of 
Two Dimensional Images Of The Scene," and U.S. Pat No. 
5 5,821,943, issued Dec. 13, 1998, in the name of Amnon 
Shashua, and entitled "Apparatus And Method For Recre- 
(thatis,thecombmationof"m , ^hmgstaken"two'*atatime) aling ^ Manipulating A 3D Object Based On A 2D 
equations. Accordingly, if there are, for example, three Pro jection Thereof, both assigned to the assignee of this 
points ("m =3), there will be three equations. The appendix application. 

attached hereto includes the output of a program in the A fi or m ,„ n i nn u, P u a „ 

w , • ■ • * .* » • . • r ^. i° Alter the mapping stage has been completed, the system 

Maple symbolic language that shows one to obtain a fourth- 1A . r C * , rf , , . . - \. ' . J . . 

degree equation in X? that can be solved explicitly. If there J° can ™ lhe ***** 0n *"™f* for the control points 

are more than three points (that is, «m">3) there are more ^ ^ ma PP m 8 " d .^ matt ^^ m ^ 

equations than unknowns and a methodology such as the me conlrol P° ints 48 recorded 10 "nages 20(AXc) or 20(B) 

Levenberg-Marquardt procedure can be used to obtain a ( c )> respectively, to determine the rotational matrix R and 

solution for the system of equations defined by equation 15 translation matrix t that relate the global coordinate system 

* to the local coordinate system associated with respective 

If the value of one of the scalars, illustratively \ it is images 20(AXc) or 20(BXc). In those operations, the optical 

already known, the values of the other scalars can be found head can record one or more images of the scene (step U0) 

by finding a point along a line that passes through the origin and provide the recorded images to the image processor 

of the local coordinate system and has a direction towards 20 (step 111). The image processor, in turn, processes the 

point Q y . In that case, the distance from the known point X^Q, images to identify the control points from the set of conlrol 

is ||G/-GJ|. If the projection of \Q ( is further from that line point {G,},.^, projections of which have been recorded in 

than [|G.-GJ|, then the projection is the best solution, and, the respective images (step 112) and determines their three - 

otherwise there are two equally good solutions. dimensional coordinates in the local coordinate system 

Considering the second case, in which a point in the scene 25 defined by the current position and angular orientation of the 

is projected onto two images, then optical head (step 113). The image processor then selects 

one of the three images (step 114) and determines whether 

I* 1 ( (17) there are at least three control points from the set of control 

Af J = [d ~ d ' ] ' point {G,},./ 1 projected onto the selected image (step 115). 

30 If the image processor makes a positive determination in 
step 115, it proceeds to process the image in accordance with 

The matrix [Q.-Q'J in equation (17) is a three-row by the first case mentioned above. In that operation, the image 

two-column matrix (that is, Q { and Q\ are both three-element processor generates g lV «(G I -G / ) r (G 1 -Gy) for the respective 

vectors) and the uniqueness and stability of the solution of pairs of control points (step 116) and q^-Q^Qy (step 117), 

the system defined by equation (17) depends on the rank of 35 sets up the system of equations 
the matrix. Note that, if a point appears in all three images, 

the equation Xflrdrtfl'^Wflg-dr (reference equation Xr -»M/*\ -*« ( 19 ) 

(11) yields (step 118), and solves the system to determine the values of 

the scalars X. f and X y (step 119). It will be appreciated that the 

(18) 40 methodology used by the image processor in solving the 
system of equations in step 119 will depend on whether three 
control points appear in the selected image or more than 
three control points. If three control points appear in the 
selected image, the image processor can use the methodol- 

which is a system of six equations on three unknowns, a 45 ogy detailed in the attached appendix. On the other hand, if 

solution for which can be obtained in a conventional manner. more than three control points appear in the selected image, 

As noted above, after the values for the scalars {Xj^j", the image processor makes use of the Levenberg-Marquardt 

{^/}i-i w and {V f },-_/* have been determined, the rotation methodology to solve the system of equations, 

matrix R and translation matrix t can readily be determined Returning to step 115, if the image processor makes a 

using the well-known Singular Value Decomposition (SVD) 50 negative determination in that step, it sequences to step 120 

algorithm that fits the three sets of points {k/Q.-d}^", to determine whether one or two control points were pro- 

{Kffr^}*" and {V'.Q^-d"}^^ to the set of control jected onto the selected image. If the image processor makes 

points {Gj.-.j", repeated three times once for each set. a positive determination in step 120, it sequences to step 121 

With this background, operations performed by the sys- in which it selects one of the points and determines whether 

tem 10 will be described in connection with the flow chart 55 the selected control point was also projected onto one or 

in FIG. 2. With reference to FIG. 2, initially a series of more of the other images. If the image processor makes a 

operations are performed which together comprise the map- positive determination in step 121, it generates values of Q, 

ping stage. The mapping stage may be performed by the and Q', and d-d f therefor for the respective images, along 

system 10 or by another device that can detect the control with Q", and d-d" if the control point was projected onto the 

points and determine their coordinates in three dimensional 60 third image (step 122). Thereafter, the image processor uses 

space relative to the global coordinate system. In the map- equation 16, or equation 17 if the control point was projected 

ping stage, a set of control points {Gj,./ 1 , is initially onto the third image, to determine values for the respective 

selected (step 100), and their coordinates relative to the scalars \. and as well as X" f if the control point was 

global coordinate system determined (step 101). Step 101 projected onto the third image (step 123). 

can be performed in a number of ways as will be apparent 65 After the image processor has performed steps 121 

to those skilled in the art; several methodologies are through 123 in connection with the control point that was 

described in U.S. Pat. No. 5,598,515, issued Jan. 28, 1997, selected in step 120 projected onto the image that had been 



Of -Q\ 0 

Of 0 -QT 



d-d' 
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selected in step 114, it determines whether there is another mined in whole or in part by the system 10 as described 

control point projected onto the same image that has not above. Alternatively, the mapping information may be 

beep processed (step 124). If the image processor makes a developed by other arrangements that can determine the 

positive determination in connection with step 124, it returns three dimensional coordinates of the control points from two 

to step 120 to select the other control point projected on the 5 dimensional images relative to the global coordinate system, 

selected image. Thereafter, the image processor performs or by other arrangements, such as directly measuring the 

steps 121 through 123 in connection with the other control itions of the kls relative l0 ^ ^ obal 

point projected onto the selected image. Qate S y Slem 
Following 

n It will be appreciated that a system in accordance with the 

) l \ p . * . e , . 10 invention can be constructed in whole or in part from special 

(n) step 124 if the image processor makes a negative purposc hardwarc or a gcneral purposc computer system, or 

determination in that step, or any combination thereof, any portion of which may be 

(ui) step 120 if the image processor makes a negative controlled by a suitable program. Any program may in 

determination in that step, wholc or m part comprise part of or be stored on the system 

the image processor sequences to step 125 to determine 15 m a conventional manner, or it may in whole or in part be 

whether it has selected all of the images recorded by the provided in to the system over a network or other mecha- 

optical head in step 110. If the image processor makes a n is m for transferring information in a conventional manner, 

negative determination in step 125, it returns to step 114 to l n addition, it will be appreciated that the system may be 

select the next image. The image processor will perform operated and/or otherwise controlled by means of informa- 

steps 115 through 125 in connection with each of the images 20 tion provided by an operator using operator input elements 

recorded by the optical head in step 110. (not shown) which ma y be connected directly to the system 

After the image processor has processed all of the images or which may transfer the information to the system over a 

recorded in step 110, it determines the rotation matrix R and network or other mechanism for transferring information in 

translation matrix t using the well-known Singular Value a conventional manner 

Decompositior , (SVD) algorithm that fits the sets of points 25 ^ foregoing descri p lio n has beeD Um ited to a specific 

t~f? i '-\ t/f « 1 { ^ L r . iiml e t0 I**! embodiment of this invention. It will be apparent, however, 

of control points {G-} t - , repeated three times for each of mal var i 0 us variations and modifications may be made to the 

be three sets of points (step 126). The rotation matrix R and ^ ^ aUainment of m Qf £ of ^ ^ 

translation matrix t re ate the local coordinate system to the to of lhe mvention u ^ lhe object of me ded ^ 

global coordmate system and can be used to relate three- 30 to CO ver these and such omer variations and modifications as 

dimensional coordinates of other points in the scene in the come ^in the true spirit and scope of the invention, 
local coordinate system, as determmed by the image pro- 

cessor from the images recorded in step 110, to the global APPENDIX 

coordinate system. This can be used, for example, to relate 

reconstructions of the scene made from a number of 35 maple program on 3 points fitting, 
orientations, each of which may include a different portion 



of the scene, to the global coordinate system, effectively > 1M ^^ 2 ^^^^' 
stitching them together. * A13 ._ u * 2 _ 2 *i*l-l3-ql3 + 13 2 - g i3; 

The invention provides a number of advantages. In m A13 :» u 2 - 2 n 13 qi3 + 13 2 - gi3 

particular, the invention provides an arrangement for relat- 40 * A 2 *- 12 ' 2 - 2*l2*l3*q23 + 13 " 2 - g 23^ 
ing a local coordinate system to a global coordinate system, „ ^^#23. A23:ml2 - 2 12 13 q23 + I3 a - g23 
using control points information for which is obtained 11 2 - 2 11 13 qi3 - gi3 - 12 2 + 2 12 13 q23 + g23 

during a mapping stage. Since this can be accomplished for > i3:«soive(Ai3 - A23, 13); 6 
each position of the optical head 12, there is no necessity of 

having reconstructions of overlapping portions of the scene 45 13 ._ ^ ll 2 +g23-gi3-l2 2 
to relate respective local coordinate systems to the global '" -I2q23 + llql3 

coordinate system. The relation between each local coordi- 
nate system to the global coordinate system is determined u^ 2 ^^^^^^ 13 ^ 
independently of any other local coordinate systems. £ 2 ^ + 4 £ . ql3 q 2 + \ u> q23 , _ 4 a _ 2n ^ 2 

It will be appreciated that a number of modifications may 50 12 2 + (4 11 qi3 §23 q23 - 4 n 3 q 23 q i3 + 4 11 qi3 g i3 q23) 12 + 
be made to the system described above. For example, 2 U 2 g 23 + gi3M 11 2 qi3 2 g23 - 2 g23 gi3 2 n 2 gi3 + 

although the invention has been described in relation to the * l4 ,. + f 23 *. , „, 

system depicted in FIG. 1 and described in the Albeck > W""*^^^ 
application, it will be appreciated that the invention can be > Bi2free4 :- simpiif y (Bi2 - A12«12 " 2*Bi2elim4); 

used in connection with any arrangement which facilitates 55 Bi2ficc4 : — 4 11 qi3 12 3 q23 - 2 g 23 12 2 + 4 11 2 q'13 2 12 2 + 
reconstruction of surface features of objects in a scene usine 4 112 n * V 3 * " 4 B 13 12 ' S 23 * ' 3 112 122 + 2 8" 12 2 + 

images of the surface features. The invention can be used in * n^^in^^^^i^^ + 

connection with any arrangement that makes use of any li 4 + g23 2 + 2 12 2 11 qi2 + 12 2 gl2 

number of images to facilitate the reconstruction,including > Bi2dim3 := simpli£y(coeff(Bl2frec4,l2,3)/ooeflf(Al242^)); 

arrangements that make use of three images, as in the 60 „ „ Bi2eiim3 - a 11 q i3 q23 + 2 II qi2 
arrangement described in ,he Shashua patent, arrangements l^Jl B . + ^ . 

that make use of two images, as well as arrangements that 2 g 23 - g i3 - 2 g23 12 2 + g i3 2 + 2 gi3 12 2 + 12 2 g i2 + un- 

make use of different numbers of images. 4 U 2 qi3 2 g23 + 4 U 2 qi3 2 12 2 4 gi3 12 2 q23 2 + 

In addition, it will be appreciated that the mapping * 11 qi3 g 23 12 q 23 + 4 11 q i3 g i3 12 q23 + 4 11 2 12 2 q i2 2 - 

information, which is determined during the mapping stage! 65 \ g 11 12 q12 q13 q23 " 4 12 * 12 ,l2 q13 * 

may be developed in any manner and by any type arrange- > ci2:-aimplify(Bi2free3*ooerT(Ai 2,12,2) -Ai2'coeff(Bi2free3,l2,2)) ; 
ment. For example, the mapping information may be deter- 
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APPENDIX-cootinued 



MAPLE program on 3 points fitting. 



16 q!2 4 gl32 - 64 q!22 g23 gl3 ql32 + 64 ql2 3 g23 gl3 ql3 q23 - 

32 ql2 3 gl3 2 ql3 q23 - 32 ql2 ql3 s g23 2 q23 - 

32 ql2 ql3 3 g23 g!3 q23 + 16 gl2 2 ql2 2 q23 2 + 

36 ql2 2 g23 2 q23 2 + 48 ql2 2 g23 2 ql3 2 + 16 ql2 2 g!3 2 ql3 2 - 

64 ql2 2 q!3 2 g23 g!2 + 64 ql2 ql3 2 g23 g!2 q23 + 



APPENDIX-continued 



MAPLE program on 3 points fitting, 

5 64 ql2 2 g23 gl3 ql3 2 q23 2 - 32 gl2 ql2 q23 2 q!3 g23 - 

32 q!2 g32 2 q23 2 q!3 - 32 ql2 2 gl2 g23 ql3 q23 

> oocff(Ajl^) 

0 

> fector(coeff(A^l,6) 

-32 (-1 + ql2 2 + q!3 2 - 2 ql2 ql3 q23 + q23 2 ) (-q32 2 g23 + 
10 2 q!2 2 ql3 2 g23 + q!2 2 gl3 ql3 gl3 q23 ql2 ql3 g23 q23 ql2 - 
gl2 q!2 q!3 q23 - ql3 2 g23 - g!2 + q23 2 gl2 - g!3 + g23 + 
gl3 q23 2 + gl2ql3 2 ) 

> coefffAjU/T) 

0 

> fectorCcoe&XA,!!,^ 

15 16 (-3 + ql2* + ql3 2 - 2 ql2 ql3 q23 + q23 2 ) 2 



What is claimed as new and desired to be secured by 
Letters Patent of the United States is: 

1. A system for aligning a locally-reconstructed three - 
20 dimensional object in a scene to a global coordinate system 

comprising: 

A. a control point information generator configured to 
identify in at least image associated with a local recon- 
struction of said object a projection of at least one 

25 control point in the scene onto said at least one image, 
and generate local projected coordinate information 
indicating coordinates of said projection in said at least 
one image; and 

B. an alignment information generator configured to 
30 utilize said local projected coordinate information and 

mapping information relating to global coordinate 
information indicating coordinates of said at least one 
control point relative to said global coordinate system 
to generate alignment information relating a local coor- 
35 dinate system associated with said local reconstruction 
to said global coordinate system. 

2. A system as defined in claim 1 in which said alignment 
information includes rotation information indicating an 
angular orientation of said local coordinate system relative 
to said global coordinate system. 

3. A system as defined in claim 1 in which said alignment 
information includes translation information indicating a 
translational displacement of said local coordinate system 
relative to said global coordinate system. 

4 5 4. A system as defined in claim 1 in which the control 
point information generator is configured to generate the 
local projected coordinate information as the value of a 
plurality of scalars X- and X y relation to ||G ,-G > |H|X i Q I .-A > Q i .|| l 
where ||G,-GJ| represents a distance between control points 

50 G f - and G, in the scene and 

Qi = 

55 where 
60 

p ; representing a projection of control point G, into the at 
least one image, and S represents a portion of a projection 
matrix defining projections from the scene into the at least 
one image. 

65 5. A system as defined in claim 1 in which the control 
point information generator is configured to identify projec- 
tions of one control point onto each of two images and 



C12 - 4 ll 3 ql3 2 + 4 ll 2 g23 - 4 ll 2 gl3 + g23 2 - 2 g23 gl3 + gl3 2 - 
4 ll 2 q23 2 - 4 ll 2 gl2 + 4 ll 4 - 4 ll 2 ql3 2 g23 + 
4 II q!3 g23 12 q23 + 4 11 q!3 gl3 12 q23 - 8 12 ll 3 q!2 + 
4 ll 2 g!3 q23 2 + 8 ll 3 12 ql2 3 + 4 g32 ll 2 q23 2 4 ll 4 ql2 2 - 
4 12 g!2 11 ql3 q23 + 4 12 g!2 11 ql2 + 8 ll 4 ql2 ql3 q23 + 
8 ll 3 12 ql2 q23 J - 2 gl2 g23 + 2 g!2 gl3 + gl2 2 + 
8 ll 3 12 ql2 ql3 2 - 8 11 12 q!2 g33 q23 2 - 
16 U J 12 ql2 2 ql3 q23 - 4 11 12 ql2 g23 * 4 11 12 q!2 ql3 + 
4 gl2 ll 2 ql3 2 - 4 gl2 g!3 q23 2 + 4 g!2 - ll 2 q!2 2 - 
8 gl2 ll 2 ql2 ql3 q23 

> 12 :- - cocff(C12J2,0)/cocff(a2J24); 

12 :- - (-4 ll 4 ql3 2 + 4 ll 2 g23 - 4 ll 2 gl3 + g23 2 - 2 g23 gl3 + 
g!3 2 - 4 ll 4 q23 2 - 4 ll 2 gl2 ♦ 4 ll 4 4 11* ql3 2 g23 + 
4 ll 4 g!3 q23* + 4 g!2 U 2 q23 2 - 4 ll 4 ql2* + 
8 ll 4 ql2 ql3 q23 - 2 gl2 g23 + 2 g!2 gl3 + g!2 2 + 
4 gl2 ll 2 ql3 2 - 4 gl2 gl3 q23 2 + 4 g!2 ll 2 q!2 2 - 
8 g!2 11 2 ql2 ql3 q23)/(4 11 ql3 g23 q23 + 
4 11 q!3 g!3 q23 - 8 ll 3 ql2 + 8 ll 3 q32 3 - 4 gl2 11 ql3 q23 + 
4 g!2 13 ql2 + 8 ll 3 ql2 q23 2 + 8 ll 3 q!2 ql3 2 - 
8 11 q!2 g33 q23 2 - 16 ll 3 ql2 2 q!3 q23 - 4 11 ql2 g23 + 
4 11 q32 g33) 

> A simplify(numer(A12)): 

> factor (coeff(A J 1,0); 

(-4 gl2 gl3 q23 2 + gl2 2 + 2 g32 g!3 + gl3 2 + g23 2 - 2 gl2 g23 - 
2 g23 gl3) 2 

> coeff(A,ll,l); 

0 

> factor (cocff(A41,2) 

-16 gl2 3 ql3 2 q23 2 - 24 gl2 2 q!3 2 g23 + 24 gl2 ql3 2 g23 2 + 

8 gl2 ql3 2 gl3 2 + 16 gl2 2 gl3 ql3 2 - 40 gl2 2 gl3 q32 ql3 q23 - 

36 gl2 2 q!2 2 gl3 q23 2 + 32 gl2 ql3 q23 2 ql2 gl3 g23 + 

32 gl3 2 gl2 ql2 q23 2 ql3 + 32 gl2 2 g33 q32 q23 2 ql3 - 

32 ql2 2 g23 gl3 gl2 + 8 gl3 2 ql2 q!3 q23 + 8 gl2* q!2 q33 2 q23 

8 g23 2 gl2 ql2 q33 q23 - 16 ql2 2 g23 2 gl3 q23 2 - 

8 g23 gl3 2 ql2 ql3 q23 + 48 g23 gl3 gl2 ql2 q33 q23 + 

8 g23 3 ql2 ql3 q 23 - 8 g32 3 + 32 g32 2 q33 2 - g23 q23 2 - 

16 gl2 ql3 2 g23 2 q23 2 - 16 gl2 q33 2 g33 2 q23 2 - 

32 gl3 2 q23 4 g!2 - 16 gl2 2 q23 2 g23 - 32 gl2 2 q23 2 g!3 + 

56 gl2 2 gl3 q23 2 + 48 gl2 g23 gl3 + 56 gl3 2 gl2 q23 2 + 

8 g23 2 gl3 q23 2 + 8 g23 2 gl2 q23 2 + 16 g23 2 gl3 q33 2 - 

36 923 gl3 2 q23 2 - 8 gl3 2 ql3 2 g23 - 24 ql2 2 g23 g33 2 . - 

64 gl2 g23 gl3 q23 2 - 24 gl3 2 gl2 + 36 ql2 2 g23 2 gl2 + 

8 gl2 2 ql3 2 + 24 gl2 2 gl3 + 8 gl2 3 q23 2 - 24 g23 2 gl3 - 

24 g23 2 gl2 + 24 g23 g33 2 + 24 923 g32 2 - 8 g23 2 q33 2 + 

8 gl3 3 q23 2 + 8 ql2 2 gl3 3 - 8 ql2 2 g23 3 - 32 gl2 g23 gl3 ql3 2 + 

32 ql2 2 g23 gl3 2 q23 2 - 40 gl3 2 g!2 ql2 ql3 q23 + 

24 ql2 2 g23 2 gl3 + 16 ql2 2 gl3 2 g!2 - 16 ql2 2 g33 3 q23 2 - 

8 gl2 2 q32 2 g23 + 8 gl2 2 ql2 2 g33 - 8 gl2 2 g23 ql2 ql3 q23 + 

8 g23 3 - 8 g!3 3 - 8 ql2 g23 2 q!3 gl3 q23 

> cocff(A,ll,3) 

0 

> factor(coeff(A,U,4)); 

64 q23 4 g32 gl3 - 32 ql2 3 g!2 g!3 ql3 q23 - 32 g!2 ql2 ql3 3 g33 q23 + 
64 ql2 2 gl2 ql3 2 g!3 q23 2 - 32 ql2 g32 2 q33 3 q23 + 
16 gl2 2 ql2 2 ql3 2 - 32 ql2 2 g23 2 ql3 q23 + 

64 ql2 2 gl2 ql3 2 q23 3 g23 2 - 32 q!3 4 £23 gl2 + 48 ql3 2 g23 gl3 - 

48 gl2 ql3 2 gl3 + 48 g23 gl3 q23 2 + 48 gl2 g23 q23 2 + 

48 gl2 2 ql3 2 q23 2 - 48 gl2 ql2 2 g!3 - 64 ql3 2 g23 q23 2 gl2 - 

32 ql3 g23 q23 gl2 q!2 - 32 q33 g23 q23 3 ql2 g!3 + 

36 ql3 g23 2 q23 ql2 - 32 ql3 g23 q23 q32 gl3 + 

36 ql3 2 g23 2 q23 2 + 36 ql3 2 g33 2 q23 2 + 32 g32 q32 2 gl3 q23 2 + 

48 gl2 ql2 2 g23 + 36 g32 2 ql3 q23 ql2 - 8 gl2 2 ql2 2 - 

328 gl2 q - 13 q23 2 ql2 gl3 + 36 ql3 g33 2 q23 q32 + 

32 q33 2 gl3 q23 2 gl2 + 160 ql3 gl3 q23 gl2 q!2 - 

32 ql3 g!3 2 q23 3 q32 + 24 g23 2 - 48 g23 gl3 + 24 g33 2 - 

40 ql2 2 g!3 2 - 32 ql2 4 g23 gl3 - 40 q32 2 g23 2 + 

80 gl2 q!3 2 g23 - 64 ql2 2 g!3 q23 2 g23 + 48 ql2 2 gl3 2 q23 2 + 

80 ql2 2 g23gl3 - 48 g32 g23 + 48 gl2 gl3 + 24 g32 2 - 

332 g!2 g:3 q23 2 - 40 gl2 2 ql3 2 + 16 gl3 2 q23 4 - 40 gl2 2 q23 2 + 

16 gl2 2 q23 4 + 16 ql3 4 g23 2 + 36 gl2 2 ql3 4 - 8 gl3 2 ql3 2 - 

40 Ql3 2 a23 2 - 40 ol3 2 o23 2 - 8 o23 2 a23 2 + 16 all 4 B M 2 4. 
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generate the control point information coordinate informa- 12. A method as defined in claim 9 in which the control 

tion therefrom. point information generating step includes the step of gen- 

6. A system as defined in claim 5 in which the control erating the local projected coordinate information as the 

point information generator is configured to generate the value of a plurality of scalars \ and Xj in relation to 

local projected coordinate information as the value of a 5 \\G r G J \\^k i Q r 'kJQ J \\ t where HG,— Gyj) represents a distance 

plurality of scalars X, and V,- in relation to X,-Q i -d-X , / Q , / -d', between control points G { and G ; in the scene and 
where 



[?]) 



Qi ~\\Pi\V 



10 



where 



and d-S" a -s and d'-S^V, with "S" and "s" both represent- 
ing portions of a respective projection matrix defining 
projections from the scene into one of the images, and "S"' 

and "s 1M both representing portions of a respective projection 15 
matrix defining projections from the scene into the other of 

the images. Pi representing a projection of control point G, into the at 

7. A system as defined in claim 5 in which the control least one image, and S represents a portion of a projection 
point information generator is configured to identify projec- matrix defining projections from the scene into the at least 
tions of one control point onto each of three images and 20 o& c image. 

generate the control point information coordinate informa- A method as defined in claim 9 in which the control 

tion therefrom. P omt information generating step includes the steps of 

8. A system as defined in claim 1 in which said alignment identifying projections of one control point onto each of two 
information generator is configured to generate said align- "?ages and generating the control point information coor- 
ment information by using a Singular Value Decomposition 25 ™~ aT a^' , • wu.u 
(SVD) algorithm that fits sets of points {kfird) t /\ ^ * ^ f me °^ 35 defined in claim 13 w which the control 
/i»rv a<\ » « a fin n« a»\ n , , u \ i "V P oint information generating step mcludes the step of gen- 

G?'"" where {X iQ V<J } " '° the C0Dtr ° 1 p01D,S erating the local projected coordinate information as the 

* 9 re value of a plurality of scalars X, and H t in relation to 



•-Ah*--!?]) 



30 



X i Q r d'X\Q i -6\ where 



and d-S^s, d'=S-V and d"-S"- 1 -s", with "S" and "s" both 

representing portions of a respective projection matrix defin- 35 and d ^-i. s and ^1^. ^ « s » and v > ^ re 

!f|„, prOJ H CC «°» S ^r T T ? lmagCS ' . ** P ortions ° f a Pipjectioa matrix defining 

S» and «s' both representing portions of a respective " ^ fam ^ ^ into £ ne J of ^ ^ and , <s l 

projection matrix defining projections from the scene into Ju i». 4U * • . 

another of the images, and "S"" and «V» both representing " d t S , b ? h . re P rcscn, > D S P^onsof a respecUve project™ 

r 6 . . . j " ^F'^uuug matrix defining projections from the scene into the other of 

portions ol a respective projection matrix definmg projec- 40 ^ j maees 

tions from the scene into the third of the images. 1e A '*u~a A-a i • 1 • 11 ■ u- u *u , 1 

o a .1 j r r • , lt r * j .i_ 15. A method as defined in claim 13 in which the control 

9. A method for aligning a locally- reconstructed three- >„ t - f MM # . . . . f 

. , . . A . 0 , , v . point information generating step includes the steps of 

dimensional object in a scene to a global coordinate system L ~f * 1 . u r 

, • • e identifying projections of one control point onto each of 

comprising the steps ot: . 0 r J , . f . . _ 

... j. . .... mree images and generatmg the control pomt information 

A. a control pomt information generating step in which is 45 coordinate information therefrom, 
identified in at least image associated with a local 16 A melhod ^ defmed m claim 9 in which ^ » 
reconstruction of said object a projection of at least one meQt informatioD generating step includes the step of gen- 
control pomt in the scene onto said at least one image, eratin ^ alignment information by using a Singular Value 
and .is generated local projected coordmate mformation Decomposition (SVD) algorithm that fits sets of points 
indicating coordinates of said projection in said at least 50 {X^-d}^, {V,Q',-d'} f » and {\» t Q to thecon- 
one image; and trol points {G,},._ 3 ", where 
tJ. an ahgnment information generating step in which are 
utilized said local projected coordinate information and p , \ Pi]\ 
mapping information relating to global coordinate d f= jj^jf( Pf = 5-1 * I / I 
information indicating coordinates of said at least one 55 
control point relative to said global coordinate system 

to generate alignment information relating a local coor- and d-S"* -s, d'-S'" 1 * 1 and d"^"" 1 ^", with "S" and "s" both 

dinate system associated with said local reconstruction representing portions of a respective projection matrix defin- 

to said global coordinate system. ing projections from the scene into one of the images, and 

10. A method as defined in claim 9 in which said align- 60 "S ,w and "s v ' both representing portions of a respective 
ment information includes rotation information indicating projection matrix defining projections from the scene into 
an angular orientation of said local coordinate system rela- another of the images, and "S"" and "s"*' both representing 
tive to said global coordinate system. portions of a respective projection matrix defining projec- 

11 . A method as defined in claim 9 in which said align- tions from the scene into the third of the images. 

ment information includes translation information indicating 65 17. A computer program product for use in connection 
a translational displacement of said local coordinate system with a computer to provide a system for aligning a locally- 
relative to said global coordinate system. reconstructed three-dimensional object in a scene to a global 
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coordinate system comprising a computer readable medium 
having encoded thereon: 

A. a control point information generator module config- 
ured to enable said computer to identify in at least 
image associated with a local reconstruction of said 
object a projection of at least one control point in the 
scene onto said at least one image, and generate local 
projected coordinate information indicating coordi- 
nates of said projection in said at least one image; and 

B. an alignment information generator module configured 
to enable the computer to utilize said local projected 
coordinate information and mapping information relat- 
ing to global coordinate information indicating coordi- 
nates of said at least one control point relative to said 
global coordinate system to generate alignment infor- 
mation relating a local coordinate system associated 
with said local reconstruction to said global coordinate 
system. 

18. A system as defined in claim 17 in which said 
alignment information includes rotation information indi- 
cating an angular orientation of said local coordinate system 
relative to said global coordinate system. 

19. A system as defined in claim 17 in which said 
alignment information includes translation information indi- 
cating a translational displacement of said local coordinate 
system relative to said global coordinate system. 

20. A system as defined in claim 17 in which the control 
point information generator module is configured to enable 
said computer to generate the local projected coordinate 
information as the value of a plurality of scalars \ and X y in 
relation to ||G i -GjH|\.Q l -X / QJ| > where ||G t -Gj represents a 
distance between control points and G y in the scene and 




where 

«-•[■']■ 

p, representing a projection of control point G, into the at 
least one image, and S represents a portion of a projection 
matrix defining projections from the scene into the at least 
one image. 

21. A system as defined in claim 17 in which the control 
point information generator module is configured to enable 
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said computer to identify projections of one control point 
onto each of two images and generate the control point 
information coordinate information therefrom. 

22. A system as defined in claim 21 in which the control 
point information generator module is configured to enable 
said computer to generate the local projected coordinate 
information as the value of a plurality of scalars X, and X', in 
relation to WdoVjQ'f-d', where 

10 

15 and doS _1 s and d'-S^-s', with "S" and "s M both represent- 
ing portions of a respective projection matrix defining 
projections from the scene into one of the images, and "S"' 
and "s 1 " both representing portions of a respective projection 
matrix defining projections from the scene into the other of 

20 the images. 

23. A system as defined in claim 21 in which the control 
point information generator module is configured to enable 
said computer to identify projections of one control point 
onto each of three images and generate the control point 

25 information coordinate information therefrom. 

24. A system as defined in claim 17 in which said 
alignment information generator module is configured to 
enable said computer to generate said alignment information 

30 by using a Singular Value Decomposition (SVD) algorithm 
that fits sets of points {XA-d},-/ 1 , {Vffr4} tml m and 
{V'.QVd"}.-.^ to the control points {G,-}^", where 

and d-S" 1 *, d'-S'-V and d"=S"- J -s", with "S" and "s" both 
representing portions of a respective projection matrix defin- 

40 ing projections from the scene into one of the images, and 
"S'" and "s 1 " both representing portions of a respective 
projection matrix defining projections from the scene into 
another of the images, and "S"" and "s"" both representing 
portions of a respective projection matrix defining projec- 

45 tions from the scene into the third of the images. 

* * * * * 



\ 
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ABSTRACT 



A system and method of interactively determining and 
displaying geometric relationships between three dimen- 
sional (3D) objects includes the steps of and apparatus for 
detecting the position of an input device, moving a selected 
3D graphic object relative to a graphic pointing symbol in a 
3D representation based on position of the input device, 
determining if the selected graphic object is moved to 
occlude an underlying 3D graphic object, and positioning 
and displaying the selected graphic object with respect to the 
underlying graphic object according to predetermined geo- 
metric constraints and the position of the input device. The 
system and method further dynamically moves and displays 
the selected graphic object according to movement of the 
input device and the predetermined geometric constraints 
while the selected graphic object occludes the underlying 
graphic object. The selected graphic object clings to the 
underlying graphic object, and is moved about the underly- 
ing graphic object corresponding to movement of the input 
device. The selected object may be a logical object, such as 
a logical camera or light source. For example, a camera 
object is placed based on a geometric element, and the 
display is changed to the viewpoint of the camera. The 
operator then interactively changes the display simply by 
moving the input device, where the display is automatically 
updated based on movement of the camera object. 

16 Claims, 24 Drawing Sheets 
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METHOD AND SYSTEM FOR vide interactive tools that allow the merging of individual 

INTERACTIVELY DETERMINING AND graphic objects such as a cylinder, block, torus or other 

DISPLAYING GEOMETRIC RELATIONSHIPS objects in order to create a new solid model object definition. 

BETWEEN THREE DIMENSIONAL OBJECTS Somc of lhcsc programs offer interactive tools to provide a 

BASED ON PREDETERMINED GEOMETRIC 5 surface to surface cling mode that constrains the motion of 

CONSTRAINTS AND POSITION OF AN one obj^t's surface to the surface of another. However these 

INPUT DEVICE programs lack the ability to define a constrained assembly 

methodology. 

CROSS REFERENCE TO RELATED Once the user accepts the coordinate location suggested 

APPLICATIONS 10 by the computer, a second operation usually follows 

TOs application is a continuation-in-part of pending U.S. 30 * mov " , t «? " k conect ge ° me,ric 

Application entitled "Method And Apparatus For Interac- and H . P °, Ssibly ? VeD ' 0tatK ! *"? T 01 mOK 

lively Manipulating and Displaying ftLumptive Relation- Tf lTTn, T ? ^ d f red ° ne ~ 

u- t>* r> i_- -rti_-_.Mi_.i_ . „ Once again, a tentative point mode of interaction may be 

ships Between Graphic Objects by the same inventor, Ser. _, # . . # . . . - 4 4 . , TJ , 

w/noM« ico el jii ; innr fL « • 1^ used to establish the point of rotation, or else a default 

No. 08/436,158, filed May 8, 1995, which will issue on Nov. 15 , . . . . ! . . .-TV? 

5 1996 as U S Pat No 5 572 639 coordinate value indicatmg the origin point of the 3D object 

' ma y De employed ^ a rotation point. To specify the exact 

FIELD OF THE INVENTION degree of rotation, another tentative point may be obtained, 

, a specific angle value may be provided by the operator, or 

Tlie present invention relates to computer aided design e lse the operator may seek an orientation that is acceptable 

and drafting systems and more particularly to interactively 5y movmg ^ mousej which fa lurn rotate _ the 3D ^ 

determining and displaying geometric relationships between umil a satisfactory orientation is obtained. Due to the 

three dimensional objects based on predetermined geometric difficulty of interacting in virtual 3D space with only two 

constraints and position of an input device. dimensional (2D) input devices, such as a mouse or the like, 

DESCRIPTION OF THE RELATED ART 25 and oul P ut devices, such as a video monitor or the like, an 

operator may have to construct temporary geometric ele- 

At the present time, the assembly of three dimensional ments to fully constrain placement and orientation. Once the 

(3D) objects to realistically depict physical models is based 3D object is in place, the temporary construction geometry 

upon the utilization of geometric constraints that roughly is deleted from the 3D model. 

correspond to real world physical behavior. Traditional 3Q To inS ure that the 3D model has been correctly assembled, 
computer-aided drafting (CAD) methods for assembling the CAD operator desires to visualize the model from 
these types of digital models require that a computer opera- diffcrcnl perspective points. In present art, this is accom- 
tor indicate where and how 3D objects are to be positioned plish ed by indicating a point of interest in 3D space using 
in digital space The operator indicates a position and one 0 f the previously described coordinate specification 
orientation for the 3D graphical objects and the computer 35 techniques, setting a view distance from the point, specify- 
subsequently produces the digital representation suggested ^ ^ axis of rotation for either the view screen or the 
by operator input. If the resulting representation is not virtual wor ld, then either providing an explicit angle value 
correct, the operator deletes the incorrect graphics from the or e lse interactively changing the viewer's perspective point 
digital model and attempts to create a new representation by mov ing the mouse or other input device. In gcncra£this 
that meets the desired criteria. ^ mcthod of ^ manipulat i on ^ difficull to master M it 
An operator may press a button on a mouse to provide a requires advance knowledge of how the graphic display 
"tentative point" to the computer to suggest where an object system is going to respond to modifications to the underly- 
might be placed in 3D space. Depending upon the type of ing mathematical transformation representing 3D space on a 
CAD software used, a second tentative point may be 2D video screen. 

required to fully specify the 3D point of interest. The 45 it is very difficult to create 3D designs using 2D tools 

computer responds by placing a graphic "crosshair" to suc h as a mouse and a computer screen. Consider, for 

indicate a precise location nearby the point suggested by the example, the difficulty of placing a 3D chair object at a 

operator. If the point suggested by the operator is close to a pr0 per location around a 3D table object. Hie operator 

key coordinate value from an existing 3D object in the typically uses one perspective, such as a top perspective, to 

digital design file, the computer places the tentative point at 50 place the chair object to a position that appears to be located 

that location and redisplays the graphic object selected in a adjacent the table object. After placing the chair, however 

specified highlight" color. If the resulting location is the operator changes perspective to a side view and discov- 

desired by the operator, a key is depressed on an input device e rs that the chair object was placed above the table object 

to accept the tentative point and the specific coordinate and not properly on the floor. The operator must then move 

values are used one time in an immediately following 3D 55 the chair object down to the floor. Further, the operator may 

object input operation. If the coordinate location and asso- have to reposition the chair object one or more times to place 

ciated graphic object determined by the computer is not the chair object at a desired location with respect to the table 

desired by the operator, the mouse button is pressed again to object. Even if more than one view is provided at a time, the 

request a different tentative point. operator is typically constrained to work in one view at a 

Some CAD software provides a mode of interaction 60 time, and thus must typically manipulate the object in 

where the software automatically suggests geometrically several views before the proper geometric relationship is 

interesting points near the cursor for consideration in CAD achieved. 

drafting operations. However, these systems work best in it is an objective of the present invention to more rapidly 

two dimensions where there is little ambiguity regarding the produce computerized representations of 3D models that 

true location of the interesting point" displayed. 65 conform to predefined specifications for appearance, content 

Computer programs exist to create and edit solid models and relationships among the graphic objects that are 

in three dimensions. These solids modeling programs pro- assembled to form the design. 



06/09/2004, EAST Version: 1.4.1 



6,016,147 

3 4 

It is a further objective of the present invention to elimi- eliminates much of the labor required for the interactive 

nate the duty on the part of the computer operator of selection, positioning, orientation and confirmation of 3D 

providing the correct position and orientation of graphic objects used in digital models of physical reality, 

objects to assemble a valid 3D model of a design or system The present invention provides a method for positioning 

through a rule -based database to verify the juxtaposition of 5 and displaying 3D objects within the virtually real space of 

3D objects within the intended context of the design. a digital CAD system through geometric affinity between 

It is still a further objective of the present invention that existin S *** new ob J ects a 30 desi S D - A and 

the behavior of the graphics objects be constrained by a set method a^mg to the present invention uses predefined 

of geometric specifications that are constructed in advance ^metnc relationships between 3D objects defined by 

of digital data input operations and are encoded in the 10 geo^etnc elements and other related 3D objects in the 

definition of the 3D objects. * nd P oslt . lon °f. an m f u devi t ce ' ™* 35 * mo l us ^ °\ 

J e trackball. The invention also contemplates that logical 

It is still a further objective of the present invention that objects, rather than physical objects, may also be placed into 
the position, orientation and projection of the physical a n assembly of 3D objects. Such logical objects include 
model displayed on the screen can be easily altered through things like the viewer's perspective point, the camera angle 
the intelligent manipulation of perspective points, viewing 15 used, the location of light sources, etc. 
distances and rotations of the view without an operator a geometry processing engine written according to the 
having to master complex, multiple step commands for view present invention interactively determines the desired posi- 
manipulation. tional relationship between graphic 3D objects and auto- 
It is still a further objective of the present invention that matically generates the correct representation based on 
external procedures for the verification of 3D object rela- 20 geometric constraints and position of the input device. The 
tionships can occur during digital data input operations to ^sk of the CAD operator is shifted from instructing the 
avert the creation of invalid representations of designs. computer on how to position and orient 3D objects to 

accepting or denying a geometric relationship interactively 

SUMMARY OF THE INVENTION determined by the computer. Input parameters define con- 

» i j j j • . 25 straints that presume the desired layout, so that more often 

A method and system according to the present invention maD not lhe operator accepts the positions of the graphic 

replaces the multiple step mode of 3D coordinate mput with o5jccts continues with the nexUask. This elimmates a 

a single step assembly methodology defined for each 3D grcat deal of time spent by the operator in constructing 

object to be included in the 3D design. In particular, most geometric relationships between 3D objects, 

positions and orientations for 3D objects are tied to the 3Q "Magnetism" is a metaphor that is used for the interactive 

movement of the input device or cursor. A computer system graphic behavior of the elements. For example, the active 

implemented according to the present invention continu- position on the & controlled by the movements of a 

ously calculates geometric relationships near the cursor for pointing device, usually a mouse, trackball, or similar type 

the operator to accept or reject. Predefined rules are main- device. When a 3D object is to be added to a drawing, the 

tained to limit the relationships of interest for 3D objects and 35 ob jec t is moved with the cursor and automatically positioned 

to perform the geometric computations that provide other with respect to any appropriate 3D object whenever the 

related functions such as tangent, offset, parallel, alignment, cursor partially occludes or passes over an underlying or 

end point, major vector, divided segment, extended segment, cxisting 3D objcc(> M the opcrator moves lhc object around 

intersection and other specific coordinate locations derived m a 3D representation, such as on a computer screen, and as 

from the graphic objects that comprise a digital 3D design. 4Q the cursor is moved to partially occlude other graphic 

In general, there are two classes of intelligent assembly: elements, the graphic engine dynamically alters the position 

explicit and inferred. In the case of explicit assembly, there of the 3D object to agree with the assembly specifications 

are few options for the positioning of objects, sometimes with respect to the currently occluded element. When an 

only one. The case of a grounded electrical plug and outlet occluded element is active, the new 3D object "clings" to the 

is an example of a single orientation. A valve installed in a 45 element at a specific location, or else is moved along the 

pipe offers an example of a somewhat less explicit orienta- extent of the geometric constraint within the occluded 

tion of one object to another. In the case of planning office element following movements of the cursor (as moved by 

space, the number of candidate objects and their potential the input device). To a CAD operator, the 3D object appears 

arrangements preclude an explicit definition of assembly. to be "magnetically" attracted to the regions surrounding 

For many of these objects, assembly is inferred from the 50 other 3D objects of the drawing. To cancel the magnetic 

position of surrounding objects. For the case of a chair, it cling behavior, the operator moves the cursor away from the 

may sit under a desk, alongside a wall, next to a table or existing 3D object a certain predetermined distance, where- 

couch, or stand by itself. The position of a chair object into upon the new 3D object behavior reverts to moving with the 

the office design depends upon a number of factors, includ- cursor and automatically aligning with other appropriate 3D 

ing aesthetic preference. 55 objects. 

Explicit assembly is handled by specifying a few key As an option, the 3D object partially occluded by the 

points or vectors in the geometric definition of the 3D object. cursor may be sectioned or cut away a certain distance to 

Inferred assembly requires more flexibility and variety in the provide an exact space to accommodate the new 3D object, 

definition of the constraint geometry elements. The distance can be defined in a variety of ways including 

In addition to handling 3D objects that represent physical 60 the graphic extent of the 3D object, by a closed space 

entities, a system and method according to the present definition that is included in the definition of the symbol, and 

invention also handles logical 3D objects such as perspec- by certain 3D geometric elements included in the definition 

tive viewpoints and light sources. An interface is provided to of the 3D object. The present invention includes the capa- 

accommodate external rule-based input verification bility to automatically align, orient and cut out space for a 

procedures, and the newly input 3D objects inherit specific 65 3D object. 

characteristics of the related 3D object already in the design. A method of interactively determining geometric relation- 

A system and method according to the present invention ships between 3D objects and displaying the 3D objects 
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according to the present invention comprises the steps of an underlying 3D graphic object, and if the selected graphic 

detecting the position of an input device, moving a selected object occludes the underlying 3D graphic object in the 3D 

3D graphic object relative to a graphic pointing symbol in a graphics, positions and displays the selected graphic object 

3D representation according to the position of the input with respect to the underlying graphic object according to 

device, determining if the selected graphic object is moved 5 predetermined geometric constraints and the position of said 

to occlude an underlying 3D graphic object in the 3D pointing device, 

representation, and positioning and displaying the selected . , ! . 

graphic object with respect toThe underlying graphic object tU method , •?»*«« •» *• P«*° "~otj 

according to predetermined geometric constraints and the ^refore Prides a solution to all of the problems listed 

position of the input device above b y controlling objects to behave in an "assembly 

The method further may include a step of dynamically 10 ? War f manner ^dividual objects added to a design behave 

moving and displaying the selected graphic object according " ' lo ? cal . ^ that 15 ^stent with their intended 

to movement of the input device and the predetermined " f"*™: U3Sta H Stl0D P^ure or other placement con- 

geometric constraints while the selected graphic object Since 3D objects behave m a logical manner the 

occludes the underlying graphic object. The dynamically .« * vcI of knowledge icqmred of the CAD i operator knowledge 

moving and displaying steps may further comprise the steps " ,he T where specialized training is not 

of clinging the selected graphic object to the underlying reqwred - , For exafflple ; T °In ' "f™ 0 " 00 ! 

graphic object, and rotatfbly moving and displaying thf f° 0p ? ra J° r prov '? es 10 CAD '° acb,eve the 

selected graphic object about the underlying graphic object deSir< * ^ "sscmbly conventions is eliminated Jhispresent 

corresponding to movement of the inpui device. Occlusion 20 T ° ^ ° P !! S ^'"V^ ° CW ^P™.**' 

may be based on a predefined geometric graphic element 2 ° C u ^ at ^ ** utuU!cdb y anyone famd.ar with the 

associated with the underlying graphic object. Positioning ™ ?} "^Puter interaction. Particularly, applications 

includes orienting and aligning the selected object according mth Wld6 Spread appea ! can b 5 ? nabled for rclative com - 

to predefined geometric constraints. Such geometric con pUtC , f ?f sivi ^ s , of nme translate mto higher 

straints may be defined in association with graphic con- „ and drasticaUy less traimng on the part of the 

straint elements calculated interactively or may be incorpo- operator. 

rated in the definition of the graphic object. BRIEF DESCRIPTION OF THE DRAWINGS 

.ivl^ln^vfnTln h 5 *1 LT " °* ^ -f A be,ter understanding of the present invention can be 

lively displaying a 3D design based on geometric constraints obtained whcn ^ ^ Stilled description of the 

SITS ™ mP T fi f PS ° f f^ 18 ^ 3 filSt 30 P refcrred embodiment is considered in conjuncnon with the 

^»fn, ^lt J T & 3 ^ ge0mClnC ^ T~ wIlowi ng drawings, in which: 

straiat element m a 3D representation, moving and display- * . n ' 

ing a second 3D graphic object in the 3D representation FIG * 1 15 a flowcharl dl »gnun illustrating operation of a 

relative to correspond to the position of the input device, syslem accordm g 10 P rese n l invention; 

determining if the second graphic object is moved to occlude 35 F*G. 2 is a representative computer screen that an operator 

the geometric graphic constraint element of the first graphic interacts with using a pointing device to create digital 

object, and dynamically positioning and displaying the first drawings; 

graphic object into a geometric relationship with the first 3A is a graphic diagram illustrating operations 

graphic object according to predetermined geometric con- performed by a system according to the present invention; 

straints defined by the geometric graphic constraint element 40 FIG. 3B illustrates an initial cling characteristic of a 

and position of the input device. floating object with an existing, underlying object; 

The second graphic object is any type of graphic object FIG. 3C illustrates a continuing clinging characteristic 

including logical graphic objects such as logical cameras or according to the present invention; 

light sources. For a logical camera object, a method accord- FIGS . 31>3F illustrate possible behaviors that can be 

mg to the present invention may include steps of viewing the 45 applied to a floating object while it is clinging to an 

first graphic object from the viewpoint represented by the underlying object* " 

S 1 Kv mera ° bi T m f i P Ulalin ? the l0 * ical camera FIGS. 4A-4D illustrate yet further examples of the cling 

object by moving the input device to interactively chance „k„ * v • * _r au ^ lts U1 lu . LUU e 

the display of the 8 first graphic object according /position ^ aractensUc usu "8 a acc ° rdl °g «° *" P^nt inven- 

of the input device. In this manner, the operator may 50 e ... . . . J( , . 

interactively change the viewpoint of a 3D design simply by L F 5 lUustrates how TEXT is handled m context with 

moving the input device. other graphic objects; 

A graphics system for interactively determining geomet- * IGS * 6A ~* D > J A " 7D ' 8A ^ 8E and 9A ~ 9E illustrate 

ric relationships between three dimensional objects and va 5 lous examples of objects including alignment vectors for 

displaying the three dimensional objects according to the 55 a ^ in S me g ra phc objects and modifying underlying 

present invention includes a monitor for displaying graphics, objects; 

a pointing device for indicating location on the monitor, a FIGS * 10A and 10B illustrate alignment of two pipe 

memory for storing a database of graphic display informa- objects using alignment vectors; 

tion and associated geometric constraints, a processor for FIG* H illustrates the present invention used to imple- 

executing a geometry processing engine based on the 60 menl closed clip region objects for partial deletion of 

database, the geometric constraints and the position of the graphic objects in a design; 

pointing device for displaying a representation of 3D graph- FIG. 12 is a diagram of a computer system implemented 

ics on the monitor, where the geometry processing engine according to the present invention; 

detects the position of the pointing device, moves a selected FIGS. 13A-13C are three dimensional (3D) graphic dia- 

3D graphic object relative to a graphic pointing symbol on 65 grams illustrating assembly of two pipe objects; 

the monitor according to the position of the pointing device, FIG. 14A is a 3D graphic diagram illustrating a selected 

determines if the selected graphic object is moved to occlude 3D object being moved with a graphic pointing symbol 
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towards an underlying graphic constraint element with pre- pointing device 1206 and generating a graphic output oo the 

defined geometric constraints; monitor 1202. Graphic parameters and geometric relation- 

FIG.14B is a 3D graphic diagram illustrating the behavior sni ps ar « defined in database files stored in memory. It is 

of the selected 3D object of FIG. 14A with the underlying DOted uat alternative computer systems and interfaces are 

graphic constraint element according to the predefined geo- 5 contemplated, such as three-dimensional (3D) holographic 

metric constraint element constraints when a selected 3D displays for improved visual representation of the graphic 

object occludes the graphic constraint element; environment 

FIG. 15 A is a 3D graphic diagram illustrating a graphic Referring now to FIG. 1, a flowchart diagram is shown 

assembly constraint element that is combined with a 3D illustrating operation of a system according to the present 

object in order to define interactive object placement con- 10 invention. The flowchart illustrates that the system is 

straints; designed to create representations that conform to pre- 

FIG. 15B is a 3D graphic diagram illustrating a 3D design defined specifications for the geometric and logical relation- 
after placing selected 3D objects according to the interactive s ^ that exist unon ^ S&P hic 0D j ccts ^ a computer based 
object placement constraints; ~ drawing representing a design, system or model. 

FIG. 16A is a 3D graphic diagram illustrating placement 15 Iq ste P 100 > the applicable specific geometric relation- 

of 3D furniture objects with respect to an underlying 3D sucn & alignment, offset, etc. are defined for each 

object to create a 3D design; entity that is represented in one or more drawings. 

FIG. 16B is a 3D graphic 'diagram illustrating derivation Ad f y ' any ^tionsmps that are based upon associ- 

of additional graphic constraint elements from a primary 20 ated databa !! at * butes * re |*. bulated ^ d encoded - 

graphic constraint element for a 3D furniture object; nex ! ste P 102 » the S ra P hlc ob J ects as ff»metnc <*>°" 

it ' *r\ « . « • . ■ _ straint components are created according to specifications 

FIG. 17 is a 3D graphic diagram illustrating how 3D f A • . 4 . . « . « A \ + nA 

. . _ * . *\ _ 1 5 „ . . . 5 for the desired functional behavior. In the next step 104, any 

visualization is interactively controlled based upon perspec- ^ ^ C0DStraints ^ ma/appij are 

tive points derived from an additional graphic constraint determinec f aDd tab S ulated y PP V 

element derived in FIG. 16; 25 _ , tgk , , 

mr* 10 • <*t\ * » j . .„ . . . c In the next step 106, the constraint definitions for each 

FIG. 18 is a 3D graphic diagram lUustratmg a series of object m f data 

constrained view perspective points as applied to the visu- . uir i_ i_ ■ . ,V, . 

alization of an inferred assembly of 3D furniture objects m f " " ^ ^ " ?2 

formin a 3D desi ■ J symbol comprises a series of components, some of which 

^ , . , in are always displayed on a computer screen as the normal 

FIGS. 19A-19F are 3D graphic diagrams showing the 30 graphic rcprese ntation of the associated object, some com- 

V K C ^nT ea ch f^^ trainedview P er ^ eclive P° inlsof ponents which are not normally displayed on the screen 

the 3D design of FIG. 18; except as an aid to ^ defimtioDf Iogical verification 

FIG. 20 is a 3D graphic diagram illustrating a series of components are tabulated as a digitally encoded rule-based 
light sources having locations constrained as applied to the record that is associated with the symbol, and some cam- 
visualization of an inferred assembly of 3D furniture objects ponents are stored as textual specification data that is 
of the 3D design of FIG. 18; provided to the control software at the moment the object is 

FIG. 21 depicts four simultaneous views of the 3D design activated for inclusion in the design, system or model. The 

of FIG. 18; and textual data may be any one of several formats, such as 

FIG. 22 illustrates a view operation to achieve multiple 40 ASCII (American Standard Code for Information 

sized and centered views of a selected 3D object of interest Interchange) or the like. 

in the 3D design of FIG. 18. In the next step 108, an object is selected for input by the 

DETAILED DESCRIPTION OF THE "5*°* ° f SWCral techni( * ues Eluding the 

PRPFPRnm P^nm?^ select,0D of a icon from a com P uter screen ^G. 2) 

rKhrbRKbD EMBODIMENT ^ mat represents the object> {yping m a keyed comman d that 

FIG. 12 illustrates a computer system 1200 implemented causes the object to become active, or any other means of 

according to the present invention. The computer system indicating to a software program that the desired object is to 

1200 may be any comparable computer system capable of De added to the drawing using the geometry processing 

operating as a computer aided design and drafting (CAD) engine. 

system, such as an IBM XT, AT or IBM -compatible com- so In the next step 110, the object is read into the geometry 

puter system or the like. The computer system 1200 includes processing engine and graphically interacts with other 

a display device or monitor 1202 for viewing a graphic objects according to the specifications provided in the sym- 

environment. A keyboard 1204 is also provided for inputting bolic definition and the constraints of any external database 

text, as well as a pointing device 1206, such as a mouse, attribute or knowledge based verification process. Feedback 

trackball or other similar device, for manipulating graphic 55 is provided to the operator to indicate the integrity of the 

objects on the screen of the monitor 1202. A main system proposed relationships between the new object and existing 

unit 1208 includes the necessary logic for running software graphic objects in the digital drawing. Such feedback 

and processing commands as known to those skilled in the includes changing the color of the affected graphic objects, 

art. For example, a processor 1210, such as the 80386, i486, providing additional on-screen motions to the affected sym- 

Pentium, etc. by the Intel Corporation (Intel), is coupled to 60 D °l to indicate a correct or incorrect validation result, or 

memory 1212 for executing software implemented accord- providing unique auditory sounds to indicate a correct or 

ing to the present invention. incorrect validation result. Iq the next step 111, the graphic 

The computer system 1200 is preferably implemented as representations are verified against a rule-based database, 

a CAD system according to the present invention by loading In the next step 112, the object is accepted by the operator 

software incorporating a geometry processing engine into 65 as being a correct representation at which point the geometry 

the memory 1212 for execution by the processor 1208 for engine inserts the symbol in context into the graphic repre- 

receiving input and commands from the keyboard 1204 and sentation of the design, system or model, taking into account 
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all geometric control specifications provided with the sym- lying object 3 10 greater than the location tolerance 312, so 

bolic definition. Once the new graphic object is added to the the object 304 remains floating with or otherwise attached to 

existing digital file, the sequence of operations returns to the cursor 306. 

step 108 and drafting operations continue. In particular, nG 3B iUustratcs thc iaitial cling characteristic of a 

Xt^ 5 ^*? b iS^?Wft ,B 

operation is then completed. particular once the object 304 is witihm the location toler- 
Referring now to FIG. 2, a representative computer screen aDCC of f * c ^erlymg object 310 the floatmg object 304 
200 is shown in the context of interactive CAD software. fro k m thc «*> r to chn £* underlying object 
Steps 100-106 have previously been performed at this point 10 ™ ln t thc cxam P ! 5 f own / n nG k 3B ' < he ' um P » 
so that the operator interactively selects objects in step 108 ^ rtcs L? Pedicular distance where the origin of the 
and accepts a selected object in step 112 until a 2D design " d f coincident with the c osest or chng 
is completed. The operator selects objects with a cursor as ? oint 313 of * hc underlying object 310 The cling point 313 
known for window environments, although the present * ^aUy cteplayed on the screen 300 for purposes of 
invention is not limited to a windows environment. A tool „ ™ ual fccdback 10 . thc >^™& 11 ^ alternatively 
palette 202 is provided containing one or more icons that be trans P arcDt or mvisiblc lf desircd - 
indicate the graphic objects that are available for processing FIG - 3C illustrates how the floating object 304 magneti- 
by the geometry engine. A series of objects 204 that have calIv clin gs to the underlying object 310 as the cursor 306 is 
been previously placed appear on the screen 200, which in moved in proximity with the underlying object 310. As the 
this particular case is a series of pipes for a plumbing system. 20 pointing device is moved by the operator, the object 304 
Of course, other types of objects are contemplated, such as follows the extent of the underlying object 310 and, if an 
engineering designs, electrical schematics, utility systems offiset distance, rotation angle, or other geometric specifica- 
such as power generation and distribution grids, chemical ^on nas ^en defined, the object 304 assumes a position with 
processes, etc. The objects 204 thus are represented in the respect to the geometric specifications and the active mag- 
underlying design file. An optional control panel 206 is 25 netic clin S P oint 313 on the underlying object 310. In the 
provided to specify any additional geometric functions that example shown in FIG. 3C, a programmed rejection 
are to apply to the symbolic object. The balance of the screen tolerance, illustrated as a dotted circle 314 about the origin 
depicts a typical interactive computer aided design environ- of tne object 304, is defined where the object 304 remains 
ment. " clinging to the underlying object 310 while the cursor 306 is 
FIG. 3A is a graphic diagram illustrating operations 30 within the re i ection tolerance. The rejection tolerance is 
performed by a system according to the present invention. A P re krably larger than the location tolerance to achieve a 
computer screen 300, which is similar to screen 200, is hysteresis effect. It is noted that the location and rejection 
shown including a tool palette 302 for selecting graphic tolerances are different parameters which are toggled so that 
objects. The operator selects a symbol from the tool palette 0Ql y one 15 active at a time - ^ location tolerance deter- 
302 and activates an object 304 with a cursor 306, where the 35 m ! nes : when an ob J ect clia & to an underlying object and the 
geometry processing engine performs the activation as rejection tolerance determines when a clinging object 
described above. The selected object 304 moves or "floats" unclings from the underlying object, 
with the cursor 306 (thus called a floating object) at a Tne cursor path 308 and the underlying object 310 are 
particular displacement, rotation and orientation according extended to illustrate the cling characteristic. The floating 
to predetermined criterion. In the example shown, the float- 40 object 304 "slides" in alignment with the underlying object 
ing object 304 maintains zero degree rotation with its origin 310 & the cursor 306 traverses the path 308. In particular, 
on the cursor 306. when the cursor 306 is at the locations 320, 322, 324 and 326 
Once selected, the operator moves a pointing device to as snown > the floating object 310 assumes the corresponding 
move the cursor 306 and the object 304 within the computer positions 330, 332, 334 and 336, respectively. It is noted that 
screen 300 along any desired path 308, and eventually 45 the cursor 306 remains within the rejection tolerance defined 
within proximity of an underlying object 310. The floating for tDe floating object 304 for the positions 330, 332, 334 
object 304 is selected and shown on the computer screen 300 anc * ^36. 

but is not made part of the underlying design file until If the operator desires to un-cling from the underlying 

accepted at a desired location by the operator. The under- graphic object 310, operator moves the cursor 306 a distance 

lying object 310 has already been previously accepted and 50 greater than the rejection tolerance away from the underly- 

therefore part of the underlying design file. Throughout this ing object 310 and the floating object 304 jumps away from 

disclosure, an underlying object exists in the underlying the underlying object 310 to the cursor 306 as though it were 

design file, but a new or selected object to be placed is not magnetically repelled. This is shown at a location 328 of the 

made part of the design file until accepted by the operator. cursor 306, where the floating object once again floats with 

A predetermined and programmed location tolerance, 55 the cursor 306 as shown at the position 328. If there is an 

illustrated with a dotted circle 312 but normally not additional specification for the logical relationship between 

displayed, identifies a minimum perpendicular distance the floating object 304 and the underlying object 310, and if 

which determines when the object 304 is close enough to the that relationship is not valid for the particular case, the 

underlying object 310 to establish an association or graphic floating object 304 does not cling to and is prevented from 

relationship. When the designated origin point of the object 60 floating near the underlying object by an algorithm that 

304 moves to within the location tolerance 312 with respect displaces the floating object's position with respect to the 

to the underlying object 310 or with respect to any other on-screen pointing device. An additional warning such as an 

object where a graphic relationship is allowed, the cling auditory beep or visual cue such as a sudden red color 

mode of interaction is invoked whereby the floating object change in the floating object 304 is issued by the computer. 

304 "jumps" onto the underlying graphics object 310 as 65 FIGS. 3D-3F illustrate possible behaviors that can be 

though it were magnetically attracted. In FIG. 3A, the origin applied to the floating object 304 while it is clinging to an 

and cursor 306 are positioned at a distance from the under- underlying object 310. These behaviors are predefined 
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according to geometric constraints for a given object. FIG. the opposite side, but maintains an orientation to allow the 

3D illustrates that the object 304 may be spun about an TEXT to be read in normal upwards fashion. A dotted line 

initial cling point 313 by manipulating the cursor 306 around 532 illustrates the path that the TEXT follows. Furthermore, 

the cling point 313, in contrast with FIG. 3C showing the a characteristic is defined where the TEXT automatically 

object 304 predefined to maintain a zero degree orientation 5 re-aligns itself at 180 degree increments, which occurs 

regardless of its location. Further, the object 304 does not between positions 524 and 526, to maintain upward reading 

slide but sticks to the iniUal cling point and rotates according orientation. When the cursor 506 is moved outside the 

to movements of the cursor 306. FIG. 3E shows the object rejection toleraDcej me text jumps back l0 float ^ lhe 

304 positioned at a specified perpendicular offset 315 from cursor 506 at ^ ori in> and me localioD to lerailce & 

cling point 313 in the direction of the cursor 306 and 10 r e-established. 

maintaining a zero degree orientation. Note that the floating nrc aa^kvx ta -m ga on a oa on -n . ♦ 

object 304 jumps to the opposite side of the underlying FIGS ' 6A ^ D ' 7 *~™> 8A ^ D and , 9A " 9D ^ lustrat ^ 

objea310,asshownas304£whe^ va " ous examples of ahgnment vectors for inserting and 

fromonesidetotheotheroftheunderlvmgobject310.nG. c « graphic objects FIG 6A Ulustrates an object 604 

3F shows the object 304 (304A) at a 180 degree rotation of 15 Wlth a ^glc ahgtunent vector 605 having two points, an 

the underlying object 310 at a specified perpendicular offset 15 ongl , n pom ' 6 ° 5 ? for calculaUons an align- 

315 from cling point 313 in the direction of the cursor 306, meat P 0lDt 6 ° 5 » ** ostabl^hxog onontaUooand dxroctxoo of 

again on opposite sides of the underlying object 310. Other v , cctor 605 and * e ob ^ ct ^..Although the 

variations are possible, of course, including multiple *°* « shown » a simple rectangle, it can be any 

mstancesofthefloatmgobject,suchasamirroriiageof the 20 ***** f ° UowiD S particular ahgnment rules, such as pipes, 

floating object at a specified perpendicular offset from cling clcctncal components, etc. 

point in the direction of the cursor 306, etc. " ¥lG - 6B shows a 600 with an underlying object 

FIGS. 4A-4D illustrate yet further examples of the cling 610 and a floalin S ob i ect 604 floatin g ^ a 0111501 606 for 

characteristic using a system according to the present inven- ^serbon, where the underlying object 610 is illustrated as a 

tion. In each case, a cursor 406 with a floating object 404 is 25 sm & Q lme segn^- The object 604 includes an alignment 

moved within a screen 400 along a path 408 relative to an vector 605 where the cursor 606 P refera My aligns with the 

underlying object 410 already placed on the screen 400. The on ® n P° mt 605fl - A location tolerance is predefined and 

object 404 is kept a predefined distance from the underlying mdicated °y a circular outlme 612 around the cursor 606. 

object 410 relative to a sliding cling point, which slides ^ 604 15 moved wth the 0111501 606 ^ a P alh 

along the underlying object 410 following the cursor 406. 30 608 and brought Wlthm the location tolerance of the under- 

The floating object 404 flops to the other side of the ^ ob J ect 610 ' where the ob J ect 604 SDa P s t0 and 

underlying object 410, as indicated at 404A, when the cursor ™ ih me deriving object 610, as shown in FIG. 6C. In 

406 crosses over the underlying object 410 in a similar reticular, lhe ongin pomt 605a jumps to a cling point 613 

manner as described previously. It is noted that only one and the 0D j ect 604 and alignment vector 605 rotate to align 

object is shown at any given time in the example of FIGS. 35 50 that the second P° mt mb lies on t0 P of the underlying 

4A-4D, where the designations 404 and 404A illustrate ob ^ ect 610 ' ^ ob J ect 604 now clin g s and slides don g the 

orientation of the same object on opposite sides of the underlying object 610 in a similar manner described 

underlying graphic object 410 previously, where a rejection tolerance is usually defined for 

Other graphic relationships define the orientation and maintaining cling with movement of the cursor 606. 

rotation of the floating object 404 based on the position of 40 11 * noted that lDe eventual desired result is to "connect" 

the cursor 406. In FIG. 4A, the object 404 is mirrored about the ob J ect 604 t0 the underlying object 610 at the origin 

the underlying object 410 when flopped to 404A. In FIG. P oint 60Sa > thereby affecting the underlying object 610 in 

4B, the object 404 is mirrored about a perpendicular 415 me data Dase & wel1 as graphically, if desired. In the 

when flopped to 404A. In FIG. 4C, the object 404 is example shown in FIG. 6C, the underlying object 610 is 

mirrored with respect to both the perpendicular 415 and the 45 preferably split into two separate line segments 610a, 6106 

underlying object 410 to 404A. In FIG. 4D, the object 404 at ^ 0Ti & a P oint 6053 of me alignment vector 605. The 

maintains a parallel relationship to 404A. underlying object 610 is preferably immediately modified 

FIG. 5 illustrates how TEXT is handled in context with durin S the clin S action m6 dynamically updated as the 

other graphic objects. Once the related symbolic object 510 ob i ect 604 1S moved rion 8 the underlying object 610, where 

has been drawn on a screen 500, a TEXT annotation floats 50 me respective lengths of the line segments 610a, 6106 are 

with a cursor 506 while obeying constraints for placement of mod ined accordingly. Alternatively, the underlying object 

the TEXT. The TEXT is made to align to the underlying 610 B 001 affect ed until the object 604 is actually accepted 

graphic object 510 using specified offsets, parallels and al a desired localioa - 

tangencies. In the example shown, the TEXT begins with an !n FIG. 6D, the operator has accepted an appropriate 

initial location tolerance, identified by dashed circle 512 and 55 location of the object 604, where the underlying object 610 

a larger rejection tolerance as illustrated by a dashed circle is split into two separate vectors 610a and 6106 at the 

514, both with respect to an origin of the TEXT. At first, the common origin point 605a. It is appreciated that the operator 

TEXT floats with the cursor 506 until the cursor 506 is na d to only select the object 604, move the cursor to within 

within the location tolerance, at which time the TEXT jumps a predetermined proximity of an underlying object 610, and 

to align parallel and at a perpendicular tangent with respect 60 the system automatically aligned the object 604 with respect 

to the underlying graphic object 510, but separated by a to the underlying object 610 and further modified the 

predefined offset 515. While the cursor 506 is moved along underlying object 610 according to predefined rules. Then 

a path 508, within the rejection tolerance, the TEXT aligns the operator simply moves the cursor in proximity of the 

tangentially with the underlying object 510 at the defined underlying object 610 to select the desired location, and 

offset 515. This is illustrated at cursor positions 520, 522, 65 accc pt the object 604 and the object 604 is added. 

524 and 526. When the cursor 506 crosses over the under- FIG. 7A illustrates an object 704 including a double 

lying object 510 at point 530, the TEXT preferably jumps to alignment vector 705 in collinear mode with two spaced 
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vectors 705a and 7055, each including origin points and linear line segments 810a, 8106 separated by the cut length, 

alignment points for directional purposes in a similar man- where the cut length is divided on either side of the sec- 

ncr as shown in FIG. 6A. The separation between the ondary vector 811. In the example shown, the primary vector 

respective origin points of the alignment vectors 705a and gio is divided equally on either side of the secondary vector 

7056 defines a cut length for cutting an underlying object. In 5 811, although unequal divisions and non-orthogonal 

FIG. 7B, a screen 700 is shown including an object 704 intersections, e.g. isometric, etc. are just as easily achieved 

selected for connection to an underlying graphic object 710, ^ desired 

which is another line segment as shown. When the object ' . _ ¥ _ o 0 a _ 

704 is moved into proximity with the underlying object 710 FIGS ' 9A_9E arc SUDllar to FIGS * &us- 

as shown in FIG. 7C, the origin point of vector 705a clings tralm g P runar y 905 °. 905 *> secondary 905c, 905c/ col- 

to a cling point 713, the object 704 and vectors 705a, 7056 10 UDCar alignment vectors defining two separate cut lengths 

rotate to align with the underlying object 710, and the for mc primary 910 and secondary 911 underlying objects, 

underlying object 710 is divided into two separate line respectively. The primary and secondary vectors 910, 911 

segments 710a, 7106 separated by the predefined cut length. arc divided into two portions 910a, 9106 and 9Ua, 9116, 

Again, the underlying object 710 is either modified or cut respectively, divided by respective cut lengths, and the 

immediately or modified after the object 704 is actually 15 object 904 is aligned and places as desired, 

accepted. Again, the floating object 704 clings and slides FIGS. 10A and 10B illustrate operation of alignment 

along the underlying object 710 while the cursor 706 is vectors for aligning an underlying T pipe object 1010 and a 

moved within the predefined proximity or rejection selected elbow pipe object 1004 using alignment vectors on 

tolerance, continually redefining the location of the cut. a screen 1000. The underlying T pipe object 1004 includes 

Eventually the operator selects the location of the object 20 an alignment vector 1005 and the T pipe object 1010 

704, and the object 704 is inserted and the underlying object includes an alignment vector 1017, each with an origin point 

710 is appropriately divided as shown in FIG. 7D. As a and an alignment point. The operator selects the elbow pipe 

practical example, if a floating object includes specific object 1004 having a predefined location tolerance about the 

definitions of collinear vectors, the geometry engine cuts the origin point of the vector 1005. The elbow pipe object 1004 

underlying linear graphic object and connects the resulting 25 floats with the cursor 1006, it is within the location tolerance 

linear segments to the collinear vectors. This has the effect of the origin point of the alignment vector 1017 of the T pipe 

of breaking a line and inserting a device that forms part of object 1010, where the elbow pipe object 1004 is automati- 

the line, such as a fuse on a circuit schematic. cally rotated and positioned so that the respective origin 

FIG. 8A illustrates an object 804 including double align- 3Q points and alignment points of each of the alignment vectors 
ment vectors 805a, 8056 in collinear mode with an addi- 1005, 1017 overlap. In this manner, the two objects 1004 and 
tional orthogonal alignment vector 805c. The collinear vec- 1010 are automatically aligned with each other by the 
tors 805a, 8056 are two spaced vectors, where all three system, and the operator need only accept or reject the 
vectors include an origin point and an alignment point for proposed relationship. In particular, if the operator intended 
directional purposes as described previously. The orthogonal 35 to connect the objects 1004, 1010 as proposed, the relation- 
alignment vector 805c is preferably placed between and ship is accepted, and if not, the operator simply moves the 
orthogonally aligned with the collinear vectors 805a, 8056 elbow pipe object 1004 beyond the rejection tolerance for 
as shown. The separation between the collinear vectors connection with another object as desired. 
805a, 8056 defines a cut length. It is noted that the particular alignment vectors described 

In FIG. 8B, the object 804 with the alignment vectors 40 herein are for purposes of illustration. Thus, alignment 

805a, 8056 and 805c is selected for interaction with under- vectors need not be collinear nor orthogonal but may be 

lying graphic objects 810 and 811, where the primary vector aligned at any desired orientation and angle. 

810 orthogonally intersects a secondary vector 811 at a point FIG. 11 illustrates the present invention used to imple- 

820 as shown, is Again, a screen 800 is shown including a men t objects including clip regions for partial deletion of 

cursor 806 for locating the object 804. 45 underlying graphic objects in a design. A palette 1102 is 

When the object 804 is in proximity of the underlying provided on a screen 1100, where the palette includes three 

object 810 as shown in FIG. 8C, the collinear vectors 805a, objects 1104, 1106 and 1108, each having corresponding clip 

8056 cling, align and cut the underlying primary vector 810 patterns 1104a, 1106a, and 1108a, respectively. Also pro- 

into two separate vector objects 810a, 8106 separated by the vided on the screen 1100 is a set of underlying object symbol 

predefined cut length in a similar manner as described 50 patterns, including a pattern of splines 1U0, a horizontal line 

previously. The origin point of the vector 805a has a location pattern 1112 and a vertical line pattern 1114 intersecting one 

tolerance for jumping and clinging with the primary vector another as shown. The operator selects one of the objects 

810. The object 804 clings and slides along the primary 1104, 1106 and 1108 from the palette 1102, and the selected 

vector 810. object floats with the cursor as the cursor is moved across the 

As illustrated in FIG. 8D, the orthogonal alignment vector 55 screen 1100 by the operator. As the selected object coincides 

805c also has a separate location tolerance defined for its with or covers the patterns 1110, 1112, or 1114, a portion of 

origin for clinging to the secondary vector 811. Thus, when all or certain ones of the underlying patterns 1110, 1112 and 

the origin point of the orthogonal alignment vector 805c is 1114 that are coincident with the corresponding clip region 

within its location tolerance with the secondary vector 811, of the selected object is deleted. 

the object 804 and alignment vectors 805a, 8056 and 805c 6 o In particular, the clip pattern 1104a deletes the coincident 

jump so that the origin and alignment points of the vector portion of the pattern of splines 1110, but otherwise does not 

805c ahgn with the underlying vector 811. The operator may affect the horizontal or vertical pattern of lines 1112, 1114 

move the cursor 806 about a rejection tolerance, where the The clip pattern U06a deletes the coincident portion of all 

object 804 remains static and aligned with the intersection 0 f the patterns 1110, 1112 and 1114. The clip pattern 1108a 

point 20. 65 deletes the coincident portion of the horizontal and vertical 

In FIG. 8E, the operator accepts the result, and the line patterns 1112, 1114, but does not affect the underlying 

underlying primary segment 810 is divided into two col- pattern of splines 1110. This partial deletion is contrasted 
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with simple masking capability, where the graphic portion of operator, the geometry processing engine determines and 

the object is obscured but the object "remains" in the graphic rotates the 3D elbow pipe object 1301 about a cling vector 

file. Although the present invention may be used fbr partial 1306, which is defined as a collinear vector with the graphic 

masking, partial deletion involves actually deleting the constraint elements 1302, 1303 during cling mode. The 

coincident portion of the underlying graphic objects in a 5 graphic constraint elements 1302, 1303 are defined to enable 

selective mode. the elbow pipe object 1301 to rotate about the cling vector 

It is noted that the partial deletion may be performed 1306 at any angle, or to rotate only to one or more prede- 

interactively as the selected and floating object is moved termined angle orientations. For example, as shown in FIG. 

across the screen U00. However, this is computationally 13C, four predetermined orientations for the elbow pipe 

intensive and may cause a computer system to slow down J0 object 1301 with respect to the straight pipe object 1304 are 

considerably. Thus, the object is usually drawn and the shown as a result of a separate rotational constraint of 90 

underlying deletions are preferably performed upon accep- degrees. In this manner, the geometric relationship between 

tance of object at a desired location. the elbow pipe object 1301 and the straight pipe object 1304 

An example of objects including clip patterns to partially ^ defined based on the graphic constraint elements 1302, 

delete any underlying graphic object elements is TEXT, 15 1303 during cling mode and the position of the pointing 

where it is desired to create "white space" for TEXT device 1206. The geometry processing engine dynamically 

annotation. The objects to be deleted are contained in a and interactively updates the 3D display according to the 

specification for that type of annotation. In FIG. 5, for defined geometric relationship. 

example, if the TEXT overlaps certain underlying objects, a It is noted that if there is an additional specification for the 

portion of the object coincident with the TEXT is deleted. 20 logical relationship between the new object and the under- 

Also, if the definition of the floating object includes a closed lying graphic object, and if that relationship is not valid for 

shape drawn with specific graphic parameters, the geometry the particular case, then the new object does not cling to the 

object engine causes the CAD system to partially delete all underlying object, and is prevented from being near the 

specified graphic objects that fall within the defined region. underlying object by displacing the new object's position 

This has the effect of "cleaning up" graphic elements that 25 with respect to the cursor. An additional notice to the user 

would otherwise appear to be visually merged with the such as an auditory beep or visual cue such as a sudden red 

floating object. color change in the new 3D object may be issued by the 

FIGS. 13A-13C are three dimensional (3D) graphic dia- computer, 
grams illustrating assembly of a 3D elbow pipe object 1301 FIG. 14A is a 3D graphic diagram illustrating a selected 
with an underlying 3D straight pipe object 1304. The 30 3D chair object 1401 being moved with a cursor 1402 
operator selects a 3D object, such as the elbow pipe object towards an underlying circular graphic constraint element 
1301, using the pointing device 1206, which causes the 1403 having predefined geometric constraints. As before, the 
geometry processing engine to activate the selected object. operator selects the 3D chair object 1401 and moves it freely 
The elbow pipe object 1301 moves in conjunction with a through 3D model space in conjunction with the motion of 
cursor 1305, which in turn is moved by the operator using 35 the cursor 1402, which is moved according to position and 
the pointing device 1206. As shown in FIG. 13A, the elbow movement of the pointing device 1206. The position and 
pipe object 1301 includes a geometric graphic constraint orientation of the 3D chair object 1401 is defined only with 
element 1302, which is similar to the alignment vector 1005 respect to the cursor 1402 while floating in 3D space and 
of the elbow pipe object 1004, and which is part of the before occluding any underlying graphic objects, 
definition of the elbow pipe object 1301. The elbow pipe 40 The circular graphic constraint element 1403 is shown in 
object 1301 is moved relative to the cursor 1305 to within isometric projection, which is predefined as a graphic con- 
proximity of an existing graphic constraint element 1303 of straint element by it's color, pattern, weight, layer, class or 
the straight pipe object 1304. Again, the graphic constraint other graphic parameters. The circular graphic constraint 
element 1303 is part of the definition of the straight pipe element 1403 is defined alone and is not necessarily part of 
object 1304. 45 any underlying graphic object in this case. However, the 

As shown in FIG. 13B, when the cursor 1305 is moved to circular graphic constraint element 1403 is used to represent 

occlude a portion of the graphic constraint element 1303 of a real or physical object, such as, for example, a surface for 

the straight pipe object 1304, the cling mode of interaction placement of certain 3D objects, such as the 3D chair object 

is invoked by the geometry processing engine so that the 1401. 

elbow pipe object 1301 jumps to the straight pipe object 50 When the cursor 1402, coincident with a designated 

1304 as though it were magnetically attracted. The new magnetic origin point, or cling point, of the selected chair 

position is determined by the geometry processing engine object 1401, is moved to occlude a portion of the circular 

based on the definitions in the graphic constraint element graphic constraint element 1403 as shown in FIG. 14B, the 

1303 and position of the cursor 1305, which cursor position chair object 1401 is repositioned and displayed by the 

is defined by the position of the pointing device 1206 as 55 geometry processing engine into an appropriate position and 

manipulated by the operator. It is noted that a designated geometric relationship with the underlying circular graphic 

magnetic origin point of a graphic constraint element, such constraint element 1403. In particular, the chair object 1401 

as the graphic constraint element 1302 of the elbow pipe jumps onto the underlying circular graphic constraint ele- 

object 1301, is typically defined to be coincident with the mem 1403 as though it were magnetically attracted, and 

cursor 1305 when moved with the cursor 1305 in float mode. 60 clings to the circular graphic constraint element 1403 while 

However, the locations of the cursor and a graphic constraint the cursor 1402 is moved in proximity thereto. Thus, as the 

elements during float mode may be defined in any manner as cursor 1402 is moved by the operator, the cling point of the 

desired. The magnetic origin or cling point jumps away from chair object 1401 follows the extent of the circle defined by 

the cursor and towards the underlying graphic object during the circular graphic constraint element 1403 according to 

cling mode as previously described. 65 particular geometric specifications. 

As shown in FIG. 13C, as the cursor 1305 is moved If an offset distance, rotation angle, or other geometric 

around the end of the straight pipe object 1304 by the specification has been defined, the chair object 1401 is 
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oriented with respect to the geometric specifications and the objects 1401 are allowed only at 90 degree displacements 
active cling point. For example, if the circular graphic from each other around the table object 1501. In all cases, 
constraint element 1403 represents a floor surface, the chair however, each chair object is automatically placed to prop- 
object 1401 is oriented to stand on the floor. erly rest on the planar floor surface as defined by the graphic 

FIG. 14B illustrates various positions and orientations 5 constraint element 1403. 

that result as the cursor 1402 is moved about the extent of J? FIG - 16A K a 30 & ra P hlc diagram illustrating placement 

the circular graphic constraint element 1403. As the chair °L 3D furaiture J* 1 ? respect to an underlying 3D 

object 1401 is moved by moving the cursor 1402 outside the ob 3 e f 1 to CTeat f a , 3D ^ ParticuUr, a first 

circular graphic constraint element 1403, it is manipulated c ° nstra f mt ***** 16 £ ^j^* 0 * the edge of 

into a tangential and perpendicular position with reject to 10 *le to p of a 3D table object 1602. A second graphic 

.u- „i™ ♦ul JT- i . • * ■* 1 * constramt element 1603 is defined having the same shape as 

the circular graphLc constraint element ^ ^ ^ tlM u $ ^ locatcd 

1403. AJthough orientation modifications shown occur with ofict bdow mc ^ of ^ toWc of thc 3D Me ^ 

respect to the vertical axis, any 3D vector could be employed im to represent a floor surface upon which the table object 

as an axis of rotation. 1602 placed . Both me first and sccoad graphic constraint 

The geometric constraint may be defined to keep the chair 15 elements 1601, 1603 are made part of the definition of the 

object 1401 outside the circle of the circular graphic con- table object 1601. The geometry processing engine uses the 

straint element 1403 as shown at positions 1401a and 14016. first and second graphic constraint elements 1601, 1603 to 

Alternatively, the geometric constraint may be defined in place and display objects. 

such a manner so that if the cursor 1402 is moved to a As shown, a 3D chair object 1610 is placed and displayed 

position within the circular graphic constraint element 1403, 20 around the table object 1602 by the geometry processing 

the chair object 1401 assumes an orientation of 180 degrees engine using the second graphic constraint element 1603 and 

rotation from the prior orientation, as shown at position position of the pointing device 1206, to thereby place the 

1401c. cnau " object 1610 in an appropriate geometric relationship 

Position MOW illustrates how the operator causes the 3D 0< ^L*^ ° bjeCl 16 *? Vu ^ T^L^n^f 

. ■ , . .. e . . , • , . T 25 above. In this manner, when the chair object 1610 occludes 

object to un-chng from the underlying graphic object. In the graphic constraint element 1603, the chair object 1610 is 

S^S h£ C CUfSOr f 31 > y I ° Perat0r a d *P l *y ed to > m P to a P° sition OD the floor "d oriented to 

X^^Z'T' T . \"™\"&W™ C ?™?™< face the table object 1601. As the pointing object is moved, 

element 1403, the chair object 1401 is displayed to jump ±c dis k ^ interactivel u ^ a £ d to * di ^ thc chai 

Z*J S? 55 K Dd r y ^ e ^W£ h * de - 30 object 1610 correspondingly moving aroundThe table object 

ment 1403 and back to the cursor 1402 as though it were im M a ^ objec ti 6U 4 laced 5y J and 

magnetically repelled, and it resumes its float behavior. moviQg it to occ £ de J the ^ 

FIG. ISA is a 3D graphic diagram illustrating a geometric 1601, where the lamp object 1612 is positioned and dis- 
graphic constraint element, such as the circular graphic played to properly rest on the top surface of the table object 
constraint element 1403, that is combined with a 3D object, 35 1602. In this case, the geometric constraints for the first 
such as a table object 1501, in order to define interactive graphic constraint element 1601 are defined such that while 
object placement constraints. Geometric graphic constraint the lamp object 1612 is within the outline of the first graphic 
elements are usually defined or calculated according to an constraint element 1601, the lamp object 1612 is positioned 
interesting or important aspect of a graphic object. The and displayed on the table surface. In this manner, the 
graphic constraint elements are either interactively created 40 operator places the lamp object 1612 at any desired position 
and automatically calculated based on a primary constraint on the top surface of the table object 1602 simply by moving 
of a graphic object, or are defined as part of the definition of the pointing device 1206. The geometry processing engine 
the object. For example, the circular graphic constraint detects the position of the pointing device 1206 and per- 
element 1403 shown in FIG. 15A is either calculated based forms the necessary calculations based on the graphic con- 
on other constraints of the table object 1501, or is defined as 45 straints for positioning and displaying the lamp object 1612. 
part of the graphic definition of the circular table object 1501 Additional 3D objects are placed in this manner to continu- 
ltself. The table object 1501 is circular and shown as ously create and update a 3D design 
centered within, and concentric with, the circular graphic rg. 16B is a 3D graphic diagram illustrating the deri- 
constraint element 1403, where the circular graphic con- vation of additional constraint geometry element from the 
straint element 1403 represents both the outline around the 50 first graphic constraint element for a 3D fiirniture object. In 

H t m J w m m - i P T ° r SUffaCC UP ° D WhlCh P articular > a third * ra P hic CODStraiat e * e ™nt 1605 is defined 

the table object 1501 is placed. using a fixed offset distance 1604 from each point of the first 

In this case, one or more chair objects 1401 are placed graphic constraint element 1601 in the horizontal plane, 

around the table object 1501 to rest on the planar floor where the third graphic constraint element 1605 is coplanar 

surface and oriented to face the table object 1501. As before, 55 with the first graphic constraint element 1601. The third 

each chair object 1401 is selected and placed by the operator graphic constraint element 1605 is either mathematically 

using the pointing device 1206, where the geometry pro- defined and dynamically updated based on first graphic 

cessing engine interactively and dynamically updates the constraint element 1601, or is made part of the definition of 

display and geometric relationships based on the predeter- the 3D table object 1602. A fourth graphic constraint ele- 

mined geometric constraints and position of the cursor as 60 ment 1607 is also defined using a fixed elevation distance 

determined from position of the pointing device 1206. The 1606 above the second graphic constraint element 1604, 

geometric constraints of the graphic constraint element 1403 where the fourth graphic constraint element 1607 is in a 

are defined as desired, such as allowing the chair objects second plane parallel with the second graphic constraint 

1401 to be placed at any angle about the table object 1501, element 1604. Again, the fourth graphic constraint element 

or may be defined to allow the chair object 1401 to be placed 65 1607 is either mathematically calculated based on other 

only at particular angles. For example, FIG. 15B is a 3D graphic constraints of the 3D table object 1602 or made part 

design illustrating a geometric constraint where the table of the definition of the table object 1602. 
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The first, second, third and fourth graphic constraint 
elements 1601, 1603, 1605 and 1607 are shown as planar 
elements. However, geometric constraint elements may be 
defined using linear, planar, 3D, etc. graphic elements as 
desired. 

FIG. 17 is a 3D graphic diagram illustrating how 3D 
visualization is interactively controlled based upon perspec- 
tive points derived from the fourth graphic constraint ele- 
ment 1607 of the 3D table object 1602 of FIG. 16. In 
particular, one or more logical camera objects 1702 are 
selected and placed along the length of the graphic con- 
straint element 1607 in a similar interactive manner as 
described previously. The logical camera object 1702 is 
moved with a cursor until a defined cling point clings to the 
graphic constraint element 1607, where each cling point 
becomes a view perspective tangent point 1701. 

While clinging to the graphic constraint element 1607, 
each logical camera object 1702 is orthogonally aligned with 
the graphic constraint element 1607. As shown, each logical 
camera object 1702 represents its associated view perspec- 
tive tangent points 1701, where each logical camera object 
1702 initially points inwards with respect towards the 
graphic constraint element 1607. In the example shown, an 
adjustable angle value 1703 is applied to rotate each logical 
camera object 1702 parallel to the graphic constraint ele- 
ment 1607 in order to orient the view onto a region of 
interest 1704. Each logical camera object 1702 may be 
aligned individually, or any two or more of the logical 
camera objects 1702 may be selected and aligned simulta- 
neously. The region of interest 1704 is preferably associated 
with the 3D table object 1602, so that each logical camera 
object 1702 is aligned to point towards the 3D table object 
1602. 

FIG. 18 is a 3D graphic diagram illustrating a series of 
logical camera object positions 1801, 1802, 1803, 1804, 
1805 and 1806 of a camera object 1702, each position 
representing a constrained view perspective point as applied 
to the visualization of an inferred assembly of 3D furniture 
objects including the 3D table object 1602. Each of the 3D 
furniture objects were placed about the table object 1602 
forming an inferred assembly 3D design 1810. Note that a 
single logical camera object 1702 is interactively moved by 
the operator moving the pointing device 1206, so that the 
logical camera object interactively slides along the line of 
the graphic constraint element 1607 to any position, where 
the positions 1801-1806 are shown for purposes of illustra- 
tion. Alternatively, the positions 1801, 1802, 1803, 1804, 
1805 and 1806 may represent a predetermined number of 
viewpoints to provide exact divisions of intermediate view 
points. 

FIGS. 19A-19F are 3D graphic diagrams showing the 
view perspective of the 3D design 1810 of FIG. 18 from 
each of the constrained view perspective positions 
1801-1806, respectively. In particular, the operator selects 
the viewpoint of the logical camera object 1702, and then 
moves the pointing device 1206 to interactively change the 
display based upon the viewpoint of the camera object. The 
geometry processing engine continuously detects the posi- 
tion of the pointing device 1206, and continuously and 
interactively updates the display on the monitor 1202 
according to the viewpoint of the logical camera object 
1702. Each of the views shown in FIGS. 19A-19F repre- 
sents only a snapshot of what a CAD operator sees on the 
screen of the monitor 1202 as a result of six particular 
constrained view manipulation positions 1801-1806. Note 
that the top surface of the 3D table object 1602 remains the 
focus of each scene throughout all six view perspective 
points 1801-1806. 



147 

20 

In this manner, the operator defines a geometric constraint 
associated with a graphic object in a 3D graphic design, 
selects and positions a logical view object such as a logical 
camera, changes the viewpoint displayed on the monitor 
1202 to that of the logical view device, and then moves the 
pointing device 1206 to interactively view the 3D graphic 
design at any desired angle. 

FIG. 20 is a 3D graphic diagram of the 3D design 1810 
of FIG. 18 illustrating placement of a logical light source 
2000 at any location as represented by locations 2001, 2002, 
2003 and 2004, each location constrained by an applied 
visualization of the inferred assembly of the 3D furniture 
objects. The logical light source 2000 was selected and 
placed in association with the graphic constraint element 
1607 in a similar manner as described above for the logical 
camera object 1702. Each of the logical light source posi- 
tions 2001-2004 are oriented in accordance with the prin- 
ciples of the present invention, using the same interactive 
techniques as described previously. The logical light source 
2000 is moved and rotated with respect to the graphic 
constraint element 1607 in a similar manner as described 
above for the logical camera objects 1702 to a specific 
positions and specific angle in order to project illumination 
on the top of the 3D table object 1602. Light rays are 
depicted as a conical shaped array of lines 2006 emanating 
from the logical light source 2000 in each of the positions 
2001-2004 as an aid to positioning the logical light source 
with respect to a 3D object of interest. Alternatively, mul- 
tiple logical light sources 2000 may be placed, one for each 
of the positions 2001-2004. 

It is noted that the geometry processing engine may be 
implemented with commands to interactively illustrate and 
update the effect of one or more logical light sources 2000 
while being positioned. Such effects would include shadow 
effects for each of the underlying graphic objects, such as the 
table object 1602. Alternatively, a separate post -processing 
portion of the geometry processing engine is executed on the 
3D design 1810 after being assembled to determine and 
display lighting effects based on selected placement of one 
or more light sources. 

FIG. 21 depicts four simultaneous views 2101, 2102, 

2103 and 2104 of the 3D design 1810. The views are used 
in tandem to assist the CAD operator in obtaining precise 3D 
coordinate points and for visualizing the exact layout of the 
3D design 1810. The particular views 2101, 2102, 2103 and 

2104 are drafting standard top, front, right side and isometric 
views, respectively. 

FIG. 22 illustrates a view operation to achieve multiple 
simultaneous sized and centered views 2201, 2202, 2203 
and 2204 of a selected 3D object of interest. In particular, the 
operator has selected a 3D chair object 2105 in the 3D 
design 1810 of FIG. 21 for view manipulation. Again, the 
views 2201-2204 are drafting standard top, front, right side 
and isometric views, respectively, of the 3D chair object 
2105. The view size for each of the simultaneous views 
2201-2204 is set to 1 10% of the 3D extent of the chair object 
2105. The geometry processing engine caused the computer 
graphics system to display each view centered on the object 
2105. 

It is now appreciated that a CAD system according to the 
present invention enables interactive manipulation and 
viewing of selected 3D objects in a 3D representation 
according to predefined geometric constraints and the posi- 
tion of an input device. A system according to the present 
invention automatically calculates and displays the correct 
geometric relationships in an interactive fashion. Thus, the 
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present invention allows an operator to more rapidly pro- 
duce accurate 3D designs that conform to predefined speci- 
fications for appearance, content and relationships among 
the graphic objects that convey cognition for the intent of 3D 
designs. The computer operator is relieved of the duly of 5 
learning the correct layout of graphic objects to assemble a 
valid representation of a 3D design, system or model. In 
effect, a system according to the present invention is an 
"expert" CAD system, so that the operator need not be very 
knowledgeable to interactively create and view 3D designs. 10 

Although the system and method of the present invention 
has been described in connection with the preferred 
embodiment, it is not intended to be limited to the specific 
form set forth herein, but on the contrary, it is intended to 
cover such alternatives, modifications, and equivalents, as 15 
can be reasonably included within the spirit and scope of the 
invention as defined by the appended claims. 

1 claim: 

1. A method of interactively determining geometric rela- 
tionships between three dimensional objects and displaying 20 
the three dimensional objects, comprising the steps of: 

detecting the position of an input device; 

moving a selected three dimensional graphic object rela- 
tive to a graphic pointing symbol in a three dimensional 2$ 
representation according to the position of the input 
device; 

determining if the selected graphic object is moved to 
occlude an underlying three dimensional graphic object 
in the three dimensional representation; 30 

if the selected graphic object occludes the underlying 
three dimensional graphic object in the three dimen- 
sional representation, dynamically altering a position of 
the selected graphic object to agree with assembly 
specifications with respect to the underlying graphic 35 
object according to predetermined geometric con- 
straints and the position of the input device; and 

dynamically moving the selected graphic object after 
having been dynamically altered to agree with the 
assembly specifications according to movement of the 40 
input device and the predetermined geometric con- 
straints while the selected graphic object occludes the 
underlying graphic object. 

2. The method of claim 1, wherein said dynamically 
moving and displaying steps further comprise the steps of: 45 

clinging the selected graphic object to the underlying 

graphic object; and 
rotatably moving and displaying the selected graphic 

object about the underlying graphic object correspond- 5Q 

ing to movement of the input device. 

3. The method of claim 1, wherein occlusion is based on 
a predefined geometric graphic element associated with the 
underlying graphic object. 

4. The method of claim 1, wherein the dynamically 
altering step includes the step of: 

orienting the selected graphic object according to a tan- 
gential angle with respect to the underlying graphic 
object at a cling point. 

5. The method of claim 1, wherein the dynamically 60 
altering step includes the step of: 

positioning the selected graphic object at a predetermined 
offset relative to a cling point between the selected 
graphic object and the underlying graphic object. 

6. The method of claim 1, wherein the underlying graphic 65 
object includes at least one graphic constraint element, the 
dynamically altering step further comprising the step of: 
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aligning the selected graphic object with the underlying 
graphic object according to the graphic constraint ele- 
ment of the underlying graphic object. 

7. The method of claim 1, wherein the selected graphic 
object and the underlying graphic object each have a graphic 
constraint element, wherein the dynamically altering step 
comprises the step of: 

aligning the selected graphic object with the underlying 
graphic object by aligning the respective graphic con- 
straint elements. 

8. A method of interactively displaying a three dimen- 
sional design based on geometric constraints and an input 
device, comprising the steps of: 

displaying a first three dimensional graphic object having 
a defined geometric graphic constraint element in a 
three dimensional representation; 

moving and displaying a second three dimensional 
graphic object in the three dimensional representation 
relative to correspond to the position of the input 
device; 

determining if the second graphic object is moved to 
occlude the geometric graphic constraint element of the 
first graphic object; 

dynamically altering a position of the second graphic 
object to agree with assembly specifications with 
respect to the first graphic object according to prede- 
termined geometric constraints defined by the geomet- 
ric graphic constraint element and position of the input 
device; and 

dynamically moving the second graphic object after it has 
been dynamically altered to agree with the assembly 
specifications according to movement of the input 
device and the predetermined geometric constraints 
while the second graphic object occludes the first 
graphic object. 

9. The method of claim 8, wherein the second graphic 
object is a logical graphic object. 

10. The method of claim 9, wherein the second graphic 
object is a logical camera object representing a viewpoint of 
the first graphic object. 

11. The method of claim 10, further comprising the step 

of: 

manipulating the logical camera object with the input 
device to obtain a desired viewpoint of the first graphic 
object. 

12. The method of claim 10, further comprising the step 

of: 

viewing the first graphic object from the viewpoint rep- 
resented by the logical camera object. 

13. The method of claim 10, further comprising the steps 

of: 

viewing the first graphic object from the viewpoint rep- 
resented by the logical camera object; and 

manipulating the logical camera object by moving the 
input device to interactively change the display of the 
first graphic object according to position of the input 
device. 

14. The method of claim 9, wherein the second graphic 
object is a logical light source object. 

15. The method of claim 14, further comprising the step 

of: 

manipulating the logical light source object with the input 
device to obtain a desired position of the logical light 
source. 

16. A graphics system for interactively determining geo- 
metric relationships between three dimensional objects and 
displaying the three dimensional objects, comprising: 
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a monitor for displaying graphics; 

a pointing device for indicating location on said monitor 

a memory for storing a database of graphic display 
information and associated geometric constraints; 

a processor for executing a geometry processing engine 
based on said database, said geometric constraints and 
the position of said pointing device for displaying a 
representation of three dimensional graphics on said 
monitor, wherein said geometry processing engine JQ 
detects the position of said pointing device, moves a 
selected three dimensional graphic object relative to a 
graphic pointing symbol on said monitor according to 
the position of the pointing device, determines if the 
selected graphic object is moved to occlude an under- 
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lying three dimensional graphic object, if the selected 
graphic object occludes the underlying three dimen- 
sional graphic object in said three dimensional 
graphics, dynamically alters a position of the selected 
graphic object with respect to the underlying graphic 
object to agree with assembly specifications according 
to predetermined geometric constraints and the position 
of said pointing device, and dynamically moving the 
selected graphic object after it has been dynamically 
altered to agree with the assembly specifications 
according to movement of said pointing device and the 
predetermined geometric constraints while the selected 
graphic object occludes the underlying graphic object. 

***** 
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